I'm pretty sure that if you take a burlap sack and start smacking a stone statue with it, the statue is going to be just fine whilst your burlap sack is going to get shredded after a while.
It's an improvised weapon. It does improvised weapon damage.
If a non-magical whip can kill someone, then it should surely be possible for an old lady to beat you down with her purse.
I see where you are going, though, and I disagree. Magic is magic, yes, but it's also specific to what it is doing;
The problem comes with weapons that don't have a bonus to attack or damage. In every previous edition, the reason why something required a +1 sword to hit was because the +1 represented how it was magically sharper and more damaging than a non-magical weapon. Your weapon must be +X more powerful than a regular weapon in order to hurt this enemy.
In 5E, that's no longer the case. You can have a sword that
isn't sharper or more damaging than a regular non-magical sword (like a Flametongue or a Frostbrand), and it will still hurt an iron golem that is otherwise entirely immune to non-magical sources of physical damage. It doesn't matter that its specific enchantment is unrelated to its sharpness or damage capacity; it's just that it has magic on it
at all that somehow allows it to hurt an iron golem.
Not that drinking a potion would make
you inherently magical, the same way that enchanting a suit of armor would make the armor magical. (Although I
could imagine a DM ruling otherwise.)