I think the main trick here is to factor in the variety of strength in different building materials, of different ages and qualities.
Most walls should NOT be made of reinforced stone or solid hardwood, except where you actually plan on someone trying to break down the walls, such as a dungeon or the like.
Your average dwelling wall should be closer to the strength of a cheap modern wall, whether you're talking plaster or sticks. A normal human being, given that they don't accidentally hit the wood in the wall, can punch right through drywall (an incredibly liberating feeling, but not recommended for those without great health insurance, just in case you hit it wrong
). Similarly, dealing with those boards, an axe can go through them fairly quickly.
Poor joints and mortar and simple age can make things much easier to take down (or, simply, pull apart. Seriously, why punch an old brick wall when you can just chip out the half-loose mortar and take the bricks apart?
There should also be some rules for damaging yourself (or others
) via impact... I can assure you, punch a tree hard enough, you'll drop a hit point or two. Slam an orc's face into a steel bar, and I think it not-unfair for them to lose a bit of vitality.
Even if they just had a few abstract categories of wall on the quality side (Crumbling, Poor, Average, Solid), and abstract categories for type (Paper, Plaster, Brick, Wood, Stone, Metal), with a few rules for how to deal with each type...
Just make sure you throw in enough Poor Plaster walls between your Solid Wood walls and Crumbling Metal walls for Grum the Pounder to get his fun in.
--
On a related note, I once got my DM to give me a Pick of Earth Parting.
Suffice to say, the Tower dungeon he threw at me did not stand tall for long.