The physics of spells are outside the normal universe, but really this is a discussion about the physics of swift actions.
swift action
A swift action consumes a very small amount of time, but represents a larger expenditure of effort and energy than a free action. You can perform one swift action per turn without affecting your ability to perform other actions. In that regard, a swift action is like a free action. However, you can perform only a single swift action per turn, regardless of what other actions you take. You can take a swift action any time you would normally be allowed to take a free action. Swift actions usually involve spellcasting or the activation of magic items; many characters (especially those who don't cast spells) never have an opportunity to take a swift action.
Casting a quickened spell is a swift action (instead of a free action, as stated in the Quicken Spell feat description in the Player's Handbook). In addition, casting any spell with a casting time of 1 swift action (such as choose destiny) is a swift action.
Casting a spell with a casting time of 1 swift action does not provoke attacks of opportunity.
Before I continue, I will point out that the above clearly states that you cannot take two swift actions. If I were running an RPGA sanctioned event as I sometimes do, I would tell any such player attempting to use two that what they are doing is against the rules.
However, in all other cases, sometimes, the rules don't quite add up. They don't fit. Usually because when they were written, they didn't anticipate one aspect of them. Their silence on the matter especially supports this.
As a DM, I have to take the initiative to sometimes say "the letter of the law contradicts the spirit of the law. And that is ultimately, why.
But to draw the example out:
Here is another swift action:
PHBII defines a feat called "Short Haft". (P82) In it, you use a swift action to change your grip on your weapon so that you attack squares adjacent to you. It requires a swift action to undo as well.
Lets say you "Short Haft" as a swift action. Then, you move and discover that it's tactically more sound to have your weapon at reach. So, you opt to give up your ability to take any other action that round, including attack, in order to switch back to reach.
In your ruling, and per RAW, I could not.
But in all other ways, it makes sense to allow it. A person in real life could easily change their grip on a reach weapon and change back within the allotted time. The only hold up here is the strictness of the rules, which in this case, don't make any practical sense.
That's why. =)