That's very true. Unless, of course, you have a DM like me. Monsters are going to fight in the most advantageous way possible, meaning they'll try to give you disadvantage while giving them advantage. For goblins, a staple among 1st level fighters, they'll use their bonus action hide to take advantage of being unseen as much as possible. They are designed to be very stealthy ambushers, after all. Now, Goblins already have a 15 AC (surprisingly high for such a token weak enemy) and being unseen gives the party disadvantage. Now, if a 1st level sorcerer casted true strike, then ray of sickness, the damage output would be better.
It's not just goblins that causes disadvantage at low levels, some cause poison upon hit: (Dretch, Quasit, Drow, flumph, etc.)
Some become invisible, like the imp or duergar.
Some restrain, like the vine blight or the constrictor snake.
Disadvantage is something a wizard is likely to face against enemies that take advantage of their abilities. At lower levels, these disadvantage states can be frequent and turn the tides of battle.