Well the monk interpretation is out there. I suppose that lions, trolls and other creatures attacking with natural weapons take damage as well under that interpretation. Utterly ridiculous.
I remember thinking about this before (without that ludicrous interpretation) and deciding that it wasn't too bad; looking at it again (after having some more experience with level 8-10 characters, I've changed my mind. In practice, the feat would be ridiculously--not ridiculously overpowered, just plain ridiculous.
Let's take a couple examples (note--all of these examples assume a bull's strength--empowered in the last case; then again in my experience, that's pretty much standard play at level 6+):
Ftr 2/Wiz 6/Spellsword 1 using a +1 morning star against a barbarian with this feat. He hits for average damage: 10 points. (4.5+4str+1 enh). Tough as nails kicks in. The morning star takes 5 points of damage. No problem; it doesn't get past its hardness. Fealing cocky (and knowing that the barbarian doesn't have much in the way of AC), the character decides to power attack for 5. He hits. It's a crit! Dealing average damage, that's thirty points. Which means the morning star takes 15 points of damage. Goodbye morning star.
Now, that case was pretty much a best case scenario for the balance of this feat. The character is optimized for defense rather than offense. Now, let's see what happens in a different situation.
Ftr 4/Clr 3/Pal 2 using a +1 warhammer and divine might.
Round 1, he hits for average damage (4.5+5 str+3 Div Might+2 specialization+1 enh)=16 damage. The warhammer takes 8 points. Depending on whether you interpret warhammers as wood or metal hafted weapons, it may be starting to crack.
Round 2, he crits for 48 points of damage! The hammer takes 24 points and is toast even if it's a +3 metal hafted hammer.
But what about a worst case scenario?
Ftr 4/Bbn 2/Rog 3 using a +1 greatclub and raging
Round 1, he hits for average damage (5.5+13 (1.5 str)+2 spec+1 enh+3.5 sneak attack)=25 points of damage dealing 12 points of damage to the greatclub. If he hits agian with his second attack, his weapon has probably had it.
Round 2, Power attack (when you've a +9 strength bonus, it's not that hard to hit) and Crit! 61 points of damage. Massive damage save for the target and 30 points of damage to the greatclub. Bye bye weapon.
So the end result seems to be that any crit against a character with this feat is likely to destroy the weapon being wielded. However, high damage characters are likely to destroy their weapons within a couple of rounds even without crits. Fighter types facing enemies with this feat would justifiably be upset about losing the weapons they need in order to fight level-appropriate foes. And if PCs have this feat, NPCs with vulnerable weapons are likely to both have significantly less loot and pose much less of a challenge.