Yaarel
🇮🇱 🇺🇦 He-Mage
At first I wrote riding a gargantuan dragon, but edited it with the two sizes difference in mind.I think that's taken care of with the size restrictions. (And riding an elephant doesn't require grappling it, rules-wise, though I'm sure a DM could choose to do it that way.)
In any case, if the elephant is unwilling, then riding it is hostile, and requires grappling.
The size restrictions dont really make sense. Because the size that is bigger is more likely to determine the movement.
When dealing with same size, one can try to "hold on" in a way that impedes the movement of the opponent, or in other words inflicts the Restrained condition. If the creature is restrained, then moving at Slow speed might make sense.
In any case, grappling is a normal part of combat. And it should happen more frequently, if realism is of interest. It is better to describe grappling by means of combat mechanics, instead of skill mechanics.