I've found that the game runs smoother if I don't bother with the quick Diplomacy check.
I just let the roleplaying organically flow.
Organic flow is the ideal situation, but rolling a dice need not interfere with that organic flow. What I think you need to do is let the role-play flow until its reaches some sort of climax, then as you reach that climax use the dice to resolve (or heighten) the tension of the scene. At some point in a scene, you're going to reach the point where the outcome is doubtful - the NPC may or may not be offended, may or may not be emotionally moved, may or may not be intimidated, or may or may not be persuaded.
The adjudicated fortune rolls are as preferable to the DM entirely relying on his own judgment here, as there are to the DM relying entirely on coin flips. The specific features that are desirable are:
a) The players get a sense that the world is fair and reasonable, and that it's not actually just wholly governed by the DM's whim. This is particularly true if the DM can show the players that things aren't actually wholly arbitrary. Without a fortune roll, the players will get the sense that the outcome of social situations is basically preordained, and the DM is likely to overrule their specific plans because he doesn't like them.
b) The players get a sense that the NPCs are not merely avatars of the DM's will and preferences, but independent subcreation with their own feeling and opinions. This is particularly true if the DM can show that the mechanics of the fortune correspond to features of the imagined world. Without a fortune roll, player's are likely to get the sense that they aren't actually negotiating with the NPC's, but negotiating with the DM. And in many cases they'll right.
c) The players get a sense that investment in social skills will be rewarded in play. Without a fortune roll that demonstrably has meaningful results, charisma and everything related to it is a dump stat. In 1e, I use to DM much as you describe with only rare reliance on fortune rolls and only for really what you admit is "some minor point" and not for the "important concepts". The result was predictably that charisma was the least valued stat, and that players tended to rely on their own charisma (and were being unwittingly judged by me on the basis of their charisma) and the outcomes of such play were almost always my own unreflected upon preferences. It was often fun, particularly because I've always been pretty good at making interesting NPCs (or at least, my players have long so complemented me), but looking back it wasn't as great as it could have been.
d) By avoiding the fortune roll, what you are essentially telling players with poor social skills - shyness, poor self-esteem, speech impediments, autism spectrum problems, etc. - is that they aren't supposed and aren't going to be allowed to influence the game, at least in social situations. You might coax these persons out of their shell in other ways and by skilled DMing, but social situations in game are as likely to be as stressful and frustrating as they find them in the real world if what you are judging (consciously or unconsciously) is their skills rather than the character's skills. That's the reason that I most like to judge content, and leave the dice to judge style and sophistication of the delivery.
e) The dice, especially when thrown in the open for dramatic effect, represents a point of natural drama in the play. In a book dramatic tension occurs in those points where its clear fate is hanging in the balance and the reader eagerly devours the page to find out what happens. The throw of the dice in a game is an equivalent moment, the better because it can be shared. Everyone hold's their breath; the dice clatters. The players are briefly suspended in an emotional moment, and the dice when it comes to a stop is going to release a torrent of some sort of emotion - tragedy or victory is at stake.
I've also found that if a DM wants his player to speak "in character", then have the NPCs speak "in character". Take the role of the king and converse with the PCs and the players will usually respond in kind.
I pretty much always speak with NPC's "in character" unashamedly relying on my terrible accents and lame acting skills to at least mark that I'm "in character" even if they aren't as entertaining as I'd wish them to be. But I still find that I regularly have to prompt players to engage IC, because they have a tendency to switch to OOC when speaking to each other (until they are highly skilled indeed) and then continue in that mind frame when turning to address the NPC. Also many players with some prior experience have poor habits that I need to help train them out of and which they'll fall back into from time to time. The goal here is to eventually get into a 'flow' state where IC interaction can be begun in a completely natural manner and becomes its own special additional joy of play.
I find the entire concept of skill challenges to be a bit artificial because I tend to just have the players tell me what their PCs are doing and for the most part unless it is a moderately difficult task, it just happens. Telling me how they are jumping from rooftop to rooftop to chase the assassin is a no brainer if they have the strength to make the distances between buildings and I do not normally ask for Athletics checks (if the jumps are too far, then they roll). Checks occur only if needed, not as part of a x successes wins the challenge and y failures loses the challenge. There will be some rolls eventually that might determine if the PCs are successful or not, but the entire round robin skill challenge system is way too formal and artificial for my tastes.
Agreed. Not having to roll for automatic tasks is a function of a skill system that lacks a fumble mechanic. Once the players have enough ranks, they really can perform increasingly heroic tasks without need to rely on a fortune mechanic because they've achieved a 100% chance of success and as a general rule you don't roll for tasks with 100% chances of failure or success. I loath round robin skill challenges and other artificialities, particularly when they have no real drama. I could imagine situations where you would have organic skill challenges that would look superficially a lot like 4e's skill challenges, but they are situations where it just so happens that there is a fair correspondence between that structure and the structure of the challenge. An example might be, "Row a boat down the rapids.", where you treated the rapids as being a series of collective tests where each player described one thing they were doing to help the boat, and in each test if they group on the whole failed, the boat lost a hull point and was smashed after X number of failures. But in this case, clearly the structure of the imagined space is determining the mechanics, and not the other way around.
The problem that I see with rolling a lot of dice for roleplaying and skill challenge resolution is that some player is always bound to roll a 1 or 2 on the die which means that something bad always happens (minimally, a failure in a skill challenge, but maybe more).
Wait??? What? No, that doesn't follow at all. The skill monkey in my party has enough ranks at least a half dozen skills, that he never fails at all any task with DC 15 or less (which is most ordinary tasks). The 'face' in the party has like a +16 in diplomacy. Even if the player rolls a 1 or 2, usually nothing bad happens as the character on her worst day is still extraordinarily likeable and persuasive and achieves an above average result. Your conclusion follows only if the difficulty of a situation automatically scales to match the PC's skills. But that is wholly artificial. The way I look at the world is almost entirely the reverse of that. As the PC's increase in skill they begin to automatically succeed in tasks that are truly challenging, while at the same time heroic and seemingly impossible tasks they now have small chances of succeeding in. Rather than discouraging players from attempting things, this encourages them to attempt things since they rarely are worse off for the attempt than they would have been doing nothing.