Organic flow is the ideal situation, but rolling a dice need not interfere with that organic flow. What I think you need to do is let the role-play flow until its reaches some sort of climax, then as you reach that climax use the dice to resolve (or heighten) the tension of the scene. At some point in a scene, you're going to reach the point where the outcome is doubtful - the NPC may or may not be offended, may or may not be emotionally moved, may or may not be intimidated, or may or may not be persuaded.
The adjudicated fortune rolls are as preferable to the DM entirely relying on his own judgment here, as there are to the DM relying entirely on coin flips. The specific features that are desirable are:
a) The players get a sense that the world is fair and reasonable, and that it's not actually just wholly governed by the DM's whim. This is particularly true if the DM can show the players that things aren't actually wholly arbitrary. Without a fortune roll, the players will get the sense that the outcome of social situations is basically preordained, and the DM is likely to overrule their specific plans because he doesn't like them.
b) The players get a sense that the NPCs are not merely avatars of the DM's will and preferences, but independent subcreation with their own feeling and opinions. This is particularly true if the DM can show that the mechanics of the fortune correspond to features of the imagined world. Without a fortune roll, player's are likely to get the sense that they aren't actually negotiating with the NPC's, but negotiating with the DM. And in many cases they'll right.
c) The players get a sense that investment in social skills will be rewarded in play. Without a fortune roll that demonstrably has meaningful results, charisma and everything related to it is a dump stat. In 1e, I use to DM much as you describe with only rare reliance on fortune rolls and only for really what you admit is "some minor point" and not for the "important concepts". The result was predictably that charisma was the least valued stat, and that players tended to rely on their own charisma (and were being unwittingly judged by me on the basis of their charisma) and the outcomes of such play were almost always my own unreflected upon preferences. It was often fun, particularly because I've always been pretty good at making interesting NPCs (or at least, my players have long so complemented me), but looking back it wasn't as great as it could have been.
d) By avoiding the fortune roll, what you are essentially telling players with poor social skills - shyness, poor self-esteem, speech impediments, autism spectrum problems, etc. - is that they aren't supposed and aren't going to be allowed to influence the game, at least in social situations. You might coax these persons out of their shell in other ways and by skilled DMing, but social situations in game are as likely to be as stressful and frustrating as they find them in the real world if what you are judging (consciously or unconsciously) is their skills rather than the character's skills. That's the reason that I most like to judge content, and leave the dice to judge style and sophistication of the delivery.
e) The dice, especially when thrown in the open for dramatic effect, represents a point of natural drama in the play. In a book dramatic tension occurs in those points where its clear fate is hanging in the balance and the reader eagerly devours the page to find out what happens. The throw of the dice in a game is an equivalent moment, the better because it can be shared. Everyone hold's their breath; the dice clatters. The players are briefly suspended in an emotional moment, and the dice when it comes to a stop is going to release a torrent of some sort of emotion - tragedy or victory is at stake.
I don't disagree with you. I just do not happen to rely on a bunch of rolls. Having low Diplomacy skills in the group means that when I do ask for a Diplomacy roll, the group is going to fail a lot.
But there are definitely skills that come up more often at my table (perception, stealth) over others (religion, diplomacy) which could be viewed as rewarding only certain skills. But, all skills come up at some times and our group often tries to ensure that they have someone trained in the vast majority of skills. I could see a given DM in a 20 minute roleplaying session asking for 8 or 12 diplomacy rolls whereas at my table, there might only be 1 to 3. I just don't let rollplaying dominate the roleplaying (I also go way out of my way to attempt to adjudicate NPC actions based on what I think they would fairly say/do). And fortunately, we do not have many shy players at my table (my wife is probably the most quiet, but the rest of the table can be quite vocal both with me, and with each other).
But to give the players a sense of fairness, I handle it a lot simpler. Once in a while, I ask for a "high is good for the party roll". It's often not modified by anything.
Example:
Player: "I want to go to the shop and buy a few potions of healing."
DM: "Roll a "high is good for the party roll"."
Player: "12."
DM: "12 is good enough for a single potion, but that's all they have remaining."
Player: "10."
DM: "Not high enough, they have no potions."
Player: "19."
DM: "With that high of a roll, they have 4 potions available."
All of this is DM whim, but it doesn't necessarily come across as DM whim. I also do not necessarily roleplay the conversation with the shopkeeper unless either a) the player wants to do so, or b) I want to hand out other information to the group via the shopkeeper. Yes, roleplaying the shopkeeper every time will result in unexpected adds to the game, but we only get to play once every other weekend if lucky, so I tend to cut to the chase.
I pretty much always speak with NPC's "in character" unashamedly relying on my terrible accents and lame acting skills to at least mark that I'm "in character" even if they aren't as entertaining as I'd wish them to be. But I still find that I regularly have to prompt players to engage IC, because they have a tendency to switch to OOC when speaking to each other (until they are highly skilled indeed) and then continue in that mind frame when turning to address the NPC. Also many players with some prior experience have poor habits that I need to help train them out of and which they'll fall back into from time to time. The goal here is to eventually get into a 'flow' state where IC interaction can be begun in a completely natural manner and becomes its own special additional joy of play.
The simple solution is to not allow players to discuss in game stuff during a roleplaying conversation session. Sure, they can joke and such, but I just don't allow them to discuss options and tactics mid-scene. If they do, I point out that "the king is listening to their discussion".
Wait??? What? No, that doesn't follow at all. The skill monkey in my party has enough ranks at least a half dozen skills, that he never fails at all any task with DC 15 or less (which is most ordinary tasks). The 'face' in the party has like a +16 in diplomacy. Even if the player rolls a 1 or 2, usually nothing bad happens as the character on her worst day is still extraordinarily likeable and persuasive and achieves an above average result. Your conclusion follows only if the difficulty of a situation automatically scales to match the PC's skills. But that is wholly artificial. The way I look at the world is almost entirely the reverse of that. As the PC's increase in skill they begin to automatically succeed in tasks that are truly challenging, while at the same time heroic and seemingly impossible tasks they now have small chances of succeeding in. Rather than discouraging players from attempting things, this encourages them to attempt things since they rarely are worse off for the attempt than they would have been doing nothing.
I don't hand out rolls if a player is going to auto-make them. So yes, if a DC is high enough that a roll is required, than a 1 (and often a 2) is going to fail. Your assumption on my conclusion was incorrect.
Why have a roll at all if there is no chance of failure, and why have a chance of failure if nothing bad actually happens with the failure (at least minimally, time is wasted and the roll has to be made later on to succeed)?
Now, there could be skill checks where DC 15 automakes, but DC 20 gains something above and beyond, and DC 25 gains a ton. But, those skill checks tend to be a bit few and far between at my table. Yes in those cases, a roll of a 1 still makes the DC 15 check (at very high level) and nothing bad happens. But I just do not hand out "above and beyond" in a lot of cases, just because someone rolled an 18 on the die. You try to climb out of the pit with an 18 roll on the athletics D20, you climb out. It still took x amount of movement. You were merely successful, you did not do it in record time or any such thing.
