Well, it may not be insane, but it sure is silly. I'm hard-pressed to think of a single short story or book where the caster does something like this, much less every 6 seconds during a fight. Sometimes the rules lead to situations that are just wacko from a story perspective. I suspect that's an important reason why CapnZapp wants to come up with better rules for object interaction.
In general though, I can't think of any stories/films where someone was expected to wield a weapon and cast spells. It seems to be pretty specific to D&D.
I don't have an encyclopedic knowledge of all fantasy books/movies but most spellcasters either don't use weapons (Dr Strange) or use the weapon as part of the spell (Gandalf in the LOTR movies).
I don't see a problem replicating that in my game by saying that you are waving around your sword/mace/quarterstaff as the somatic component.