Ovinomancer
No flips for you!
Let's accept that the player has no responsibilities whatsoever to engage the social contract of a group RPG and try to play his character within the guidelines and expectations of the game. He is free to, and even expected to, attempt whatever his heart's desire, and it is fully the responsibility of the GM to determine if a roll is required, set the DC if so, and narrate the results. Good?The game of D&D involves sitting around with friends, pretending some imaginary stuff is going on, and thinking up interesting, fun, effective, etc ways for your PC to engage with that stuff. In that sense, it's an intellectual pastime. (Contrast, say, running or cycling, which are also things that can be fun and interesting to do with friends, but are primarily physical pastimes.)
Telling a player that s/he can't engage with the intellectual elements of the game is tantamount to telling him/her that s/he can't play the game.
If a PC has a 5 STR, the player of that PC is in no way precluded from playing the game. S/he can still sit around with his/her friends and come up with the action declarations that s/he thinks make sense for his/her character, given the ingame situation, the goals of the character, the preferences of the player, etc.
But some posters in this thread are saying that, if a PC has 5 INT, then the player of that PC has to refrain from fully engaging with the game in this fashion: for instance, that s/he is not allowed to engage in solving puzzles. Think about classic modules like Tomb of Horrors, or Ghost Tower of Inverness, or Hidden Shrine of Tamoachan, or the Caves of Chaos in Keep on the Borderlands: these modules are almost entirely puzzle-solving, in the sense that playing these modules is all about reasoning through solutions to the various improbable but challenging situations they throw up in front of the PCs.
Or think about a more contemporary module, like the first Freeport module, or Speaker in Dreams, or Heathen, or Bastion of Broken Souls, or Demon Queen's Enclave (or even the Shrine of the Kuo-Toa, which is an old module that is rather contemporary in feel). These modules also have a very big puzzle-solving dimension, as the players have to identify various factions, work out who is related to whom in what sort of way (allied, opposed, potential friend, certain foe, etc) and then make choices about how to inject their own PCs in to the situation and push it to some sort of resolution.
One characteristic of real people who aren't all that clever is that they can't do this sort of stuff very well. They can't make effective tactical or logistical choices. They aren't all that good at working out the dynamics of complex political or social situations. They make poor choices, relative to their own interests, because they are incapable of identifying and then reasoning through the relevant (though perhaps not immediately salient) consequences.
To require a player of a low-INT PC to play his/her PC in such a way is, in effect, to require him/her to not fully engage with these aspects of RPGing, which to many RPGers are at the core of the activity.
To me, it's fairly clear that this is why Moldvay relates INT to linguistic ability: it gives INT a clear mechanical role, as STR has in relation to opening doors (but, in Moldvay Basic, not to encumbrance). But there is no suggestion that the player of the low-INT fighter isn't nevertheless fully able to engage in the play of the game.
In the context of 3E, 4e or 5e, a low INT penalises certain skills and limits access to certain feats. That's the "handicap" that is imposed. Especially in 5e, the GM is also free to frame ingame possibilities, including the need for a check to find out what happens, by reference to a PC's INT.
But none of that implies that the player him-/herself has to take responsibility for limiting his/her PC in certain ways, any more than if the PC had a 5 STR.
(If all fiendishly difficult puzzles are resolved by INT checks there might be other issues with the campaign, but the low INT PC will be suitably penalised, just as is the low STR PC when it comes to weightlifting competitions.)
What's the difference between the DM telling the 5 STR character he has no chance of lifting a large boulder and telling the 5 INT character that they have no chance of solving the puzzle? In both cases, the DM has determined if a roll is required and decided that there's no chance of success given the low relevant ability scores. This seems an even more blatant denial of ability to engage in challenges than expecting a player to acknowledge and attempt to roleplay low ability scores.
Not being good at a challenge, or even intentionally downplaying your contribution in order to roleplay your character, isn't denial of participation, it's active participation just not solely directed at solving the challenge. I find your argument to be solely focused on the ability to be successful at overcoming a specific challenge, eg a riddle, but that fails to grasp the meta of the challenge. The challenge immediately before you is just one part of the challenge of the game, and the challenge of the game also include roleplaying a character not you. You say roleplaying a low INT would deny a player the ability to engage fully in a mental challenge, but you're missing that by roleplaying their low INT while engaging that mental challenge they ARE fully engaged in it, just with a different objective that 'be successful at solving mental challenges.' I would submit that if solving mental challenges is important to you, don't play a character with a 5 INT.