This kind of play, I feel, is fostered because of D&D where the overwhelming real consequence in the game is PC Death (even for NPCs). Lesser consequences are mitigated or ignored in service to the Hit Point god.
So rational (always optimal) decision making is rewarded.
This is a huge part of why I am such a cautious player myself,
and why I have put in a great deal of effort into making sure my players know that they won't have their ability to participate in the game, nor the character-story they're interested in exploring, ripped away merely because of a stupid die roll or whatever. There will always be consequences, and their characters might go through terrible,
terrible things (after all, did not Jafar teach us "There are things
so much worse than death!"? Low-quality movie oveall,
Return of Jafar, but that refrain is one of its brighter points.)
D&D taught me that if I don't play as cautiously and carefully as possible, it can and
always will piss in my cheerios, give me a wedgie, and throw sand in my eyes.
EDIT: Thus you have reports of players behaving this way or that way and the constant discussions about min/maxers.
It is why I provided another track for level progression through roleplaying decision points via Traits, Bonds, Flaws, Ideals.
Focusing on emotional struggle, character growth, playing into flaws, making hard decisions, providing other meaningful consequences that stand on their own away from the Hit Point god.
50 years and this is the best the designers can do....I guess I shouldn't complain, it is a war-game at the end of the day. But I feel they should advertise that more. War-game disguised as a Roleplaying-game. Sorry rant over.
No, I think you're bang on with this. D&D fans like to think that it's totally not that, but...the way the rules are made, it is. And it's one of the reasons why the hatred of things like Skill Challenges and other 4e design elements (like actually balanced classes) irritates the hell out of me. Those very things were
trying to make D&D be NOT "literally still just a wargame in a funny hat", and people
who claim to see D&D as something else hated them for it. It just...it boggles my mind how unreservedly
attached people are to everything being shackled to the Hit Point God, as you put it, to everything being so thoroughly about "don't mess up or everything you've worked for will be stolen from you in the blink of an eye....and often even when you DON'T mess up that'll happen too, but it DEFINITELY WILL happen if you do mess up".
Perhaps I should beg forgiveness for my own rant. This whole thing just irritates the hell out of me.