If I download mods will my game slow down?

In terms of performance, it depends.

Mods may have different standards than official content.
Those could be more simple, or more complex. So your performance while using a specific mod may not be the as when you use official content.

Will mods affect your game performance even if you are not using them? In some parts.
For example, as mods will appear in your Level/Vehicle selector, the game will take a few moments to also to show them. Talking about milliseconds here.
Obviously, the more mods you put, the more this delay will increase.