Spellcasters normally can achieve powers much greater and more versatile than just being a swordsman. And some types of them are born with magic (sorcerers and certain races). People generally aren't born good swordsmen.For the same reason nations are not ruled by the mightiest swordsmen: Just because you wield great individual power, doesn't mean everybody does what you want when you aren't standing over them with sword (or wand) drawn.
Now, if wizards see themselves as a "tribe" and band together to promote the interests of wizards against everyone else, they can take control of nations. But if they don't, they're just like big guys with swords. Every ruler needs some, and you always have to reckon with them in your plans, but you don't have to be one to rule.
And this is a systemic topic. I'm not saying that if there's just a single mage in the setting that they're destined to rule everything. More that if mages do exist, are fairly common, and have more power than non-mages, then the chances of them using their magic to take power is practically inevitable in the long run unless there are large roadblocks built into the setting that prevent this.
In the real world where magic doesn't exist, monarchs across the world throughout history have claimed to be divinely mandated to be royalty. In a setting where people can be born with magic, those claims (true or not) would be even stronger. People with more money (merchants and nobles) would be more likely to put their children through a school that can teach them magic (bard or wizard), which would give the family better tools to gain even more money and more influence around the world around them. It would have a snowball effect where more money increases arcane power for the family which would increase the family's ability to get more money and political power in the setting.