It's actually a lot easier to take advantage of the environment as a caster as well, but that's a good point with regards to overcoming it via superior GMing.
However from a game design perspective thats less than ideal and there's still precious little to represent things like disarming...