Your perception of what is occurring is off. The PC has goal X. The player is making a decision that thwarts that goal. the PC doesn't know that the action will thwart his goal because he's stupid, so his action can't be irrational. He would have to be aware that his action is contrary to his goal in order to be acting irrationally.
My point is that the PC's action is irrational in the sense that it is contrary to his/her interests. Hence, if the player is expected to declare such an action the player is being expected to play his/her PC as irrational. Which I think does not make for good gameplay, and is not demanded in any ruleset for D&D except perhaps 2nd ed AD&D.I think he was claiming that it is irrational on the player's part, not the character's, to do the thwarting action. Maybe "irrational" isn't technically correct, because you could argue that it's rational to prioritize a narrative over mechanical goals, but at least he's trying to define his terms.
Elfcrusher is right that, if you are playing in that 2nd ed style, then it may be rational for the player to play his/her PC as irrational. That's why I don't think much of the 2nd ed style!
And on the usage of the word "irrational" - I thought it was fairly clear that I meant objectively irrational in the sense of contrary to the PC's own goals, and hence not something which the PC has a reason to do, although s/he may falsely believe that s/he does have such a reason. Irrational does not have to mean knowingly irrational. The world is full of irrational behaviour, but only some of it is knowingly irrational in the sense described by Maxperson; much of it is the result of people not having a good understanding of the things that they have reason to do, given their interests and goals.
The first thing I notice about this post is that a particular approach to play - the 2nd ed, 1990s-era, players-provide-colour-and-characterisation-in-a-GM-driven-game approach - is elevated to RPGing per se, and other approaches dismissed as board gaming.Assuming that you aren't playing D&D as a board game, one of your goals is to roleplay your character, including any flaws such as stupidity for low intelligence. That means that it would be irrational not to roleplay your character as stupid, as that failure would thwart your goal.
But moving on from that: D&D doesn't have a "behavioural flaws" mechanic of the sort found in GURPS, HERO etc. The only thing that comes close is the alignment requirement for some classes in pre-4e editions. 5 INT isn't a flaw anymore than 5 STR is a flaw. It's a stat score which will impact some checks and, at least in classic D&D, will limit certain action declarations involving language learning and language use.
Where does 5e tell me that having 5 INT is a behavioural flaw? Answer: it doesn't. It tells players that they will be penalised on certain checks, and it tells GMs that they might have regard to the stat in framing or permitting certain action declarations.
Page 57 of the Basic PDF says:
Six abilities provide a quick description of every creature’s physical and mental characteristics . . . Is a character muscle-bound and insightful? Brilliant and charming? Nimble and hardy? Ability scores define these qualities - a creature’s assets as well as weaknesses.
The three main rolls of the game - the ability check, the saving throw, and the attack roll - rely on the six ability scores.
The three main rolls of the game - the ability check, the saving throw, and the attack roll - rely on the six ability scores.
Nothing in those paragraphs suggests any requirement that a 5 INT PC's lack of brilliance should manifest itself other than in the outcomes of these rolls. Which is not to say that various characterisations are precluded. It's to say that they're not mandated.
If I'm playing a 5 INT PC, that will show itself in the fact that my knowledge checks tend to fail. I don't need to refrain from attempting such checks as well - though if there is another way to deal with a situation that I'm more likely to succeed at, I might try that instead. These patterns of action declaration and resolution are what establish the character of my PC. I don't need, in addition, to deliberately make irrational choices for my PC. (That's not to say that some players might not. But they don't have to.)
From my perspective, this relates to my personal dislike for what I've called the 2nd ed AD&D style.you might have multiple goals. If another goal is "succeed at the adventure" then you might possibly have two conflicting goals. How you prioritize them is not a right/wrong choice, it's a preference, and if adhering to a narrow definition of roleplaying that requires you to pretend to be stupid is not your top priority, then yes it may be irrational to do so in some circumstances.
The sort of style I enjoy is one in which "succeed at the adventure" and "roleplaying my character" don't come into conflict. The way this is achieved is to align the adventure with my character: the adventure is some sort of goal for the PC, and the character of the PC is then demonstrated in the action declarations that are made in pursuit of that goal.
In this style, there's no need to introduce "stupid" choices for my PC to demonstrate that my PC is not an intellectual sort of character - that will already be demonstrated in the action declaration choices and the goals chosen for my PC.
To give a concrete example: at one stage in my 4e game, the player of the dwarven fighter-cleric wanted to reforge the dwarven thrower artefact Whelm into a two-handed maul Overwhelm. To do this, he drew upon the resources of the dwarven community, and the culmination of his participation in the process demonstrated his toughness, not his intellect: when the dwarven artificers were having trouble holding down the hammer as the magical energies were building up in the forge, the PC shoved his hands into the forge and physically took hold of the hammer so that the artificers could apply their tongs. This goal - of reforging a weapon - and this way of doing it - of drawing on the community resources, and of relying - at the crux - on his own physical prowess - show us the character of this PC (as a physical rather than intellectual type; as a leader rather than a loner; etc).
That same actual play link also illustrates how the superior intelligence of the invoker/wizard PC in the game manifests itself: he is the one who comes up with the legal arguments in a court case; who resolves the intellectual challenge of how to manage the arcane process of turning Whelm into Overwhelm; etc.
When you have these two patterns of play happening side-by-side, there is no need to trade off between "the adventure" and characterisation of the PC: playing the game, and pursuing the PCs' goals, also makes it clear that one is an intellectual and the other is not. There is no additional need to insist upon the player roleplaying his 8 INT as some sort of stupidty or inability to make good decisions.
Now, if the main focus of play is goals set by the GM; and if the action resolution mechanics don't provide the scope for the different PCs to manifest their differences in action declaration and action resolution; then I can see how there is pressure on the player of a low-INT character to manifest that through characterisation choices like making "stupid" choices. But that pressure is the result of a broader approach to play - what I've called the 2nd ed approach - which is not the only way to come at the game.