Unsolved Why do objects outside the draw distance affect performance?

Why do object that are outside the drawdistance affect the performance? I mean.. they are not rendered... so why is that?

I get fps boost when I turn far away objects invisible in the scene tree. but they are outside the draw distance anyways...