I guess that's simply something (rules penalties for casters) I've never seen. Then again a lot, but not all, of what I've seen has been with public games where the DMs are supposed to follow official rules. They can still add "flavor" and some people seem to really hate characters in heavy...