I feel the same way about unlimited at-will cantrips. I suppose I would have little problem with wizards throwing firebolts if they didn't hit as well as the fighter and do as much damage as them swinging their swords.
Luckily, they don't!
At 1st level, with a 16 attack stat:
Wizard w/ Firebolt: +3 to attack, d10 fire damage (5.5 avg hit)
Fighter w/ Greatsword & Great Weapon fighting style: +3 to attack, 2d6+3 reroll 1&2s damage (11.34 avg hit) [4.17 per die due to reroll]
Fighter w/ Longsword & Shield and Duelist style: +3 to attack, d8+3+2 damage (9.5 avg hit),
+2 AC
Fighter w/ Longbow & Archery fighting style:
+5 to attack, d8+3 damage (7.5 avg hit,
with more hits)
This increase at 5th and 11th when cantrips go up and fighters get more attacks. Wizards not adding their attack stat means it also increases at each ASI but only for fighters. Oh, and fighters getting extra ASIs means that they increase chance to hit faster than wizards as well.
If the game allows feats this becomes even more one-sided, as there are a lot more feats that increase weapon damage then increase cantrip damage.
In other words, cantrip damage isn't even close to fighter damage with weapons. I hope you are open minded and stand by your "I suppose I would have little problem with wizards throwing firebolts if they didn't hit as well as the fighter and do as much damage as them swinging their swords."
Add to it the scaling damage and I find most casters whipping around cantrips instead of spells most of the time.
It just feels so much like a video game. There is no more "bursts of power" from the casters. It's a the same across the board though since every fighter is throwing superiority dice, rogues are sneak attacking every round, rangers are hunter's mark/colossus slaying everything, and warlocks are hex/eldritch blasting everything.
This is part of the base math of the game. 6-8 encounters of 3-4 rounds each is 18 to 32 rounds of combat per long rest. Let's lowball and go 20. Add in one non-action spell per combat like shield or healing word that's 26 casting actions per day.
If you consider just a few spells cast outside combat (mage armor, some utility), you end up with way less spells then rounds of combat.
I think you need to be around 7th before 1/3 of your actions per day can be spells. At 11th you have 16 slots. If you cast mage armor and two other utility spells you still only have enough slots to cast for half your actions.
So the game design is what pushes using cantrips so much. And why restricting the number of cantrips really hurts characters, especially ones below the teen levels.