Sorry, for coming in late to the discussion, but I've found it quite interesting.
It seems to me that allowing the actions to be defined only in terms of time is central to the issue. If we say that swift actions take 1 unit of time and standard actions take 2 units of time, then why couldn't we do a swift actions as a standard action? Really, it seems to make sense. However, what if time is not the only variable? What if there is something else at work behind the scenes?
Lets introduce another variable called effort, for example. Then lets say that doing a swift action cost 1 unit of time and 3 units of effort, a move action costs 2 units of time and 0 units of effort, and doing standard action cost 2 units of time but only 2 units of effort. Now we can see how these types of actions cannot so easily be substituted for one another. Also, makes sense now why you cannot use a move action to attack.
I realize that these number smight not add up if you start totalling effort and time each round, then divide it back up, but I was never really that good at math. Anyway the theory seems to stand that there are likely more variables at play in the definition of a round then simply time. Therefore, allowing the RAW to remain seems the best bet to me.