The problem is that the character is still being penalized within the gameplay function of their primary ability. Especially when in 3.5 where your Halfling Barbarian has a worse chance to hit, has a smaller bonus to damage, and has a lower damage die to begin with. And hitting things, really...