Playing it like it's a game with rules to exploit for maximum advantage rather than playing it like you're developing a fantasy action story with a framework of rules to support that. I'll admit that there have always been people playing like the former, but it is the point where virtually all RPGs face challenges.
oD&D was about ruthless adventurers trying to walk off with anything that wasn't nailed down (and seeing if you could sell the nails). Combat was something you were meant to avoid if you could - and to rig as hard as possible if you couldn't. And rigging it as hard as possible meant exploting the world. If D&D can't handle this approach then it objectively can't handle the approach Gygax wrote it for.
The fantasy action stuff was added later.
Also mysteriously, 4e has few problems with people trying to bend it. Mostly because 4e has been errata'd whenever someone found an exploit. So by saying where "virtually all RPGs face challenges" you're running into problems given that there are two editions of D&D (oD&D, 1e) that expect people to do this becuase they are not about fantasy action stories, and one that allows for people to push it hard.
I agree with Neonchameleon on this issue. The idea that RPG rules break down when the player push them hard seems fairly common, but I think it is true only of a certain sort of simulationinst ruleset - namely, one which opens up points of decision-making, in action resolution and PC building, that are open to metagame exploitation by those players wanting to do something other than "develop an fantasy action story".
Runequest is not terribly vulnerable in this respect, because it has so few metagameable decision points.
4e is not particularly vulnerable in this respect, because it is not simuationinst.
Classic D&D is no more than moderately vulnerable, I think, because - played at low to mid levels, at least - the only class with real metagame power (the MU) is not all that strong, and there is a strong emphasis on adjudication outside the scope of the formal action resolution rules.
3E seems uniquely vulnerable, though, because it has at least a degree of simulationism in its rules, it is widely played in a simulationisnt fashion, but in both action resolution and even moreso PC building it is chock full of points of possible exploitation by players.
Putting in a mechanically intricate PC build system, and then telling people not to optimise it, is a recipe for your game breaking down in the hands of very many groups!
EDIT: The rival to 3E's uniqueness would be points-build games, which have the same tension between build systems and play goals. But points build games often have a player culture built up around them to handle the issues that result. And also tend to be a bit more self-conscious about the tensions they give rise to, I think.