See, I take the exact opposite viewpoint. I mean, it's absurd that someone could swing a sword all day. It's freaking heavy, and even if every roll that doesn't meet or exceed the target's AC is indeed a complete whiff that only connects with air, the hits you do get in obey that whole 'every action has an equal and opposite reaction' rule, which means eventually your arm is going to get a bit sore. It's entirely implausible to keep on swinging all day long, unless you're John Henry (in which case you'll drop dead at the end of the day, which kind of sucks) or a robot of some sort.
Meanwhile, why shouldn't casters be able to cast pew-pew magic all day? Magic is an external force that they're controlling by waving their hands about and muttering silly words, which (from personal experience waving my hands about and muttering silly words, though with sadly a lack of any notable magic effect, unless you count making people look at me funny) is far less tiring than swinging a sword about (nevermind the whole making-sure-the-other-guy-with-the-sword-doesn't-hit-you-back bit). It's not like magic itself is getting tired. And most importantly, in the immortal words attributed (possibly incorrectly) to former Marvel Comics Editor-in-Chief Joe Quesada,"It's magic, we don't have to explain it."
Meanwhile, why shouldn't casters be able to cast pew-pew magic all day? Magic is an external force that they're controlling by waving their hands about and muttering silly words, which (from personal experience waving my hands about and muttering silly words, though with sadly a lack of any notable magic effect, unless you count making people look at me funny) is far less tiring than swinging a sword about (nevermind the whole making-sure-the-other-guy-with-the-sword-doesn't-hit-you-back bit). It's not like magic itself is getting tired. And most importantly, in the immortal words attributed (possibly incorrectly) to former Marvel Comics Editor-in-Chief Joe Quesada,"It's magic, we don't have to explain it."