Let's see...
The siege engine with the highest damage, the heavy catapult, does 6d6 damage (DMG, p.100). That's an average of 21 points.
Level 1 half-orc barbarian, Str 18 (22 while raging) with a greataxe and Power Attack (for -1 to hit/+2 damage): 1d12+8 (1d12+11 while raging). That's an average of 14.5 points (17.5 while raging).
Assuming the same Strength, feats and non-magical weapon, the same character will do 18.5 (21.5 while raging)
by level 3 (using Power Attack for -3 to hit/+6 damage).
Higher levels will drastically increase this. (Level 11: Str 30 [16base+2race+2levels+4belt+6greaterrage], a +2 greataxe and power attack for -11/+22: 1d12+39. And if he uses an adamantine axe, objects with hardness ratings of less than 20 will crumble when he as much as looks at them.)
Daaamn...

And some people were afraid of level 16 monks using ki strike (adamantine) to punch through walls.

But it seems that, at these levels, you're
expected to be able to go
through many walls. (Not to forget... Level 9 wizard: teleport. Even though it's been nerfed. And there's other stuff as well.)