Think of it this way: you're not able to swing as well, getting that -2, but you're attacking a more delicate target on most enemies. Their ankles, knees, groin.... not very well protected with most armor, as it makes mobility improbable. The damage is still done, you just don't have that...