I think that for physical repeatable skills that are within the capability of the PC, an ok houserule is to state that it just takes longer. How long? A number of "original time units" * how much the original DC was missed by.
For example, if it takes one round to pick a lock and the PC misses the DC by 3, then it takes 4 rounds. The PC is still successful, but it avoids that whole constantly re-roll the dice problem.
If it takes a minute to get up the cliff and the PC has a rope that prevents him from falling, it might just take 6 minutes to get up that cliff.
That sounds like a nice, workable solution.
The reason I use the unit of time of 6 seconds is because that is how long a round is. If 10 rounds equal a 1 minute, 1 round equals 6 seconds and unlocking a lock is defined as an action that takes a round.
This means that either one is only spending 6 seconds of trying within their first roll of a failure.
The only way to raise one's skill is to go out and kill monsters. So... "I can't unlock this now even though I have unlocked similar locks in the past. Let me go murder a hundred people and come back to try again."?
Come on. That makes no sense.
Two things:
1. I think trying to pick a lock in combat and trying to pick a lock out of combat are two different tasks. In combat you're basically jamming your picks in there and hoping it pops open. Out of combat you can take more time. Just because picking a lock is an action you can take in combat doesn't mean that it's the same action that you take when you try to pick a lock outside of combat.
Of course it's up to the DM to determine such things.
2. You can look at XP for killing monsters (or overcoming challenges, or whatever) as a proxy for adventuring. It's a simple way of tracking how much adventuring you're doing, one that doesn't rely on the DM to make difficult and/or arbitrary judgement calls (e.g. "10 XP per level for good roleplaying"). If you're killing monsters, you're probably picking locks, searching for traps/secret doors, praying to your god, mastering spells, etc. You may not be doing those things, but it would be a lot of bookkeeping to keep track of every lock picked and all the rest.
Not that I'm that fond of XP for killing monsters, but it has its merits.
There really needs to be something in between "you succeed instantly" and "you are incapable of ever succeeding at this task ever.. unless you murder enough people to raise your level."
Why not have a roll of 1 equate to the true fail state in which the action simply cannot be tried again-- you broke it, the person you were trying to pester into something attacks you, you sprain your angle falling from the wall this time.
Repeated rolls need only mean the character is taking up more and more time trying to perform the task.
You could, and that would work. I don't like the extra dice rolling. I find that it takes up too much time and reduces the tension of a single die roll.
I can't tell you how many countless times I have done something where, had I done it right the first time, it really would have taken only seconds. But I failed... and I could spend several minutes trying over and over and over and over again... and you know what?... I manage to eventually do it.
That possibility ought to exist within a game. The system of "one roll ever, either instantly succeed without issue or fail and know that it is impossible for you to ever succeed at this no matter how long you spend"... that just doesn't strike me as workable.
Yes, there are plenty of skill rolls where there are nasty consequences of failing and one should be aware of this. But other times it is just a matter of how much time and emotional stamina you need to spend before completing the task.
I agree with what you're saying. I think I'm just approaching the issue from a different point of view.
I'd say that picking a lock takes 5-10 minutes (a good old fashioned Turn) and the single die roll represents all your progress and setbacks. If you fail then the lock has proven too tricky for you. I'd probably allow another attempt but this one would take up the rest of the hour (so no Short Rest for you), or whatever amount of time wandering monster checks work on.
That way you incorporate the game's use of time as a resource into the player's decision making. I like that sort of thing: "Well, you can try again, but I'm going to make a wandering monster check/you won't get a Short Rest."
Anyway, I hope you can see why someone might want to go with a "one roll, no retries" (or close to it) system.