I don't recall ever suggesting that size modifiers should apply to Pilot checks. Moreover, I think it's a really bad idea because, as you observed, it results in obscenely high penalties to larger ships. As a general rule, larger ships will have lower maneuverability, thus the turn modifiers...