I don't see it as dragging yourself; rather you're using the wall as if weightless to propel yourself.
And yes, throwing something heavy in one direction might move you slowly in the other.
But - and here's another messy question - does the Levitate's caster have any control of the facing-position-etc. of the target? For example, is the target always assumed to be upright with its feet toward the ground, or can it be flipped and-or spun by either itself, external forces, or the caster?
Put another way, how close does anyone's version of Levitate veer towards Telekinesis?
What you're describing may have been true in previous editions, I just see no reason to believe that it's the case in 5E. The spell does what it says what it does in my games, no more no less.
Specifically:
The target can move only by pushing or pulling against a fixed object or surface within reach (such as a wall or a ceiling), which allows it to move as if it were climbing. The target can move only by pushing or pulling against a fixed object or surface within reach (such as a wall or a ceiling), which allows it to move as if it were climbing. ... you can use your action to move the target, which must remain within the spell's range.
The caster can move the target, but cannot affect facing nor spin them upside down. There is no indication the target is bobbing around weightlessly, the spell is quite specific saying that you can only pull or push against a fixed object or surface.
So throwing a heavy object does nothing, the target can't be flipped or spun. It's not even particularly clear if you could "push off" a solid surface.