Simple answer: if the in-game characters, as reflected by the player's knowledge, don't realize a task is beyond them (which can often be the case) and the roll comes up 4, they don't and shouldn't know whether they failed due to bad luck or failed because they couldn't succeed. They just know - both in and out of character - they failed this time and have to try something else.
The problem with this is meta-currency. The player
knows it was bad luck when they rolled a 4. You might think they shouldn't know why, but the truth is the player knows. Depending on the scene, they might feel it is pretty stupid and resent failing due simply to bad luck.
Sometimes 95% failure becomes worth it when that 5% success comes through.
That, and despite everything the players can do to migitate the odds, the game is still at its heart a gamble. Embrace that, and it all makes far more sense.
I have no issue with 5% success, in and off itself. I take issue when the numbers say there should be 0%, and people want the "nat 20" to work anyway. If a roll of 20 can't make the numbers work for success, there should never be a roll. That is what I am saying.
Hell, if it's a choice between a 1-in-20 chance of safely jumping the gap vs a 0-in-20 chance of surviving what's chasing me on this side then I'm trying that jump every time.
But it isn't a 1-in-20, it is a 0-in-20. A roll of 20, -1 for STR 8, is 19 which fails the DC 20.
Allowing 20s and 1s a bit of latitude reflects, in a way, the extreme ends of the bell curve of success probability.
Sure, and I can see why some groups embrace that. Unless a DM is strict about when to call for checks and doesn't allow absurdly ridiculous thing, I can't get behind the 1 always failing and 20 always succeeding. I like to keep it simple: if the roll + bonuses makes it, you succeed. Otherwise you fail. I don't care if the roll is 20--if you don't have the bonuses to make the extra amount, no good.
If you've ever looked at the 1e combat matrix it gives an example of what I mean. As the target's AC increases the numeric progression (i.e. the roll) required to hit rises in lockstep - until it gets to 20, where it plateaus over several increasing AC ratings until resuming its climb to 21, 22, etc.
Great example. That natural 20 on the combat matrices (?) in 1E was to allow someone to hit higher ACs without bonuses. Once the numbers went over 20, you needed a natural 20 AND a bonus of +1 (for 21) or better (for higher numbers).
I suppose you could extrapolate that to 5E-mechanics, so if natural 20+bonus is "within 5" of the DC, then you can do it. In the jumping example above, that would be such a case. But if the DC was 25, then even the natural 20 would fail.
But this gets to a level of nuiance that many players don't care to embrace I would think.