My players finally came back with some feedback, and raised some valid points.
Some of them I could explain by quoting more recent parts of this discussion, but one of them remained:
the variation in falling damage according to size, coupled with the possibility to end up with 0 dmg when using negative modifiers on your dice creates the situation where large creatures will always take damage (1d6+2, a minimum of 3 con per 10') while smaller creatures might not get damaged at all (1d6-2', a minimum of 0 con per 10')
also, even when all size categories would use negative modifiers, the smaller creatures would have higher chances for survival from ANY height, since the chance to get 0 dmg per die would be larger.
My suggestion would be to switch to different dice (as in, use the size increase for damage die, Large creatures taking d8-2 dmg instead of d6-2, Huge creatures taking d10-2, etc.)
One of my players has started responding with his own progressions, using decreasing negative modifiers per height, but I think he's making things to complicated. To be complete, I'll post his suggestion here:
(NB: damage is per 10'):
Small(-1) Medium (0) Large(+1) at Height
1d6-5 ... 1d6-4 ...... 1d6-3 ...... 10ft
1d6-4 ... 1d6-3 ...... 1d6-2 ...... 20ft
1d6-3 ... 1d6-2 ...... 1d6-1 ...... 30ft
1d6-2 ... 1d6-1 ...... 1d6 ......... 40ft
1d6-1 ... 1d6 ......... 1d6+1 ...... 50ft