Try to think of it this way. Sure, you're constantly doing something in a combat round. Ducking, looking, weaving, talking, thinking. Maybe at the beginning of the round you decide to cast a spell you begin gathering the mental strength, divine favor, or psionic might you need to complete the spell. Maybe you even start pantomiming some ancient gestures and uttering nonsensical babble. But perhaps your spellcasting action isn't actually vulnerable to disruption (by attack/counterspell/etc) until a critical moment -- a particularly complicated gesutre, a long-winded multi-syllabic name for your deity, or whatever other flare you want to tag onto it. Perhaps that is why while this whole 6 seconds you've been "casting" your spell it was only vulnerable to interruption for a very brief window (your actual action on your actual turn on your actual initiative count)
The consequences of this house-rule with smart players are going to devolve into Delay/Ready wars that will just ultimately serve to bog down combat as everyone is now inserting a Delay between spellcasts. Congratulations, you've just halved all your spell durations.
