Think of it this way.
A giant swinging a club at a nimble monk. As opposed to a monk kicking said giant in the shins. Both cases of straight out attack and the larger size getting the penalty.
Now initate the grapple. Gotta make a touch attack, there you do have the penalty, either to the...