I suppose if it was the only thing that differentiated them, this would be true. But, speed not making a difference? Ask any group who played 4e. Or had a party that walked somewhere (probably at low levels) where time was a factor.
Also, I don't get this:
Okay, so, we're changing shorter...