First off, there's three parts in this argument I'd say, and there might be a difference in how we prioritize. The three parts I see is:
1. Realism. Even though it's an abstract system, we want some resemblence of realism. That's why a greataxe does more damage than a longspear, and it's why you can't fall fifty feet without taking damage a'la DMC3.
2. Balance/playability. It shouldn't be too easy to avoid the sneak attack, because that makes rogues bad.
3. Continuity (sorry if this is a bad word in the context; English is not my main language). We want the rules to work like the rest of the game as far as possible, so it's easy to remember. Too many extra rolls and exceptions is bad, as is rarely used functions like for example percentile dice.
I think due to the realism argument, which we are both engaging in and which should be addressed, it might be a good idea to lay down the 1-st level commoner vs. 19-th level fighter situation since we can both agree that a 19th level fighter is somewhere between a superhero and a demigod.
Let's say it's a first level warrior with a short sword versus a 5th level fighter in a non-magic fullplate. The warrior would have +2 attack bonus (+1 str, +1 bab) and the fighter AC 19 (+1 dex, +8 full plate. Let's assume he's using a two-handed weapon). The rogue would be a 5-th level rogue, but that's of less importance.
If this assumptions aren't okay with you, we can skip them, but throughout this post I will use them since 5th level is where you usually draw the line between heroes as they COULD be IRL and high-fantasy superheroes.
Except that's how it works in abstracted D&D combat. If I roll a lucky 20 twice in a row with any weapon, I've managed to find a vulnerable point in your armor, or more likely, somewhere it isn't protecting.
No, that's not how it works. It doesn't work that way as is because a mobile person is never considered helpless in RAW. You WANT it to work like that, but right now it doesn't. Yes, rolling 20 twice in a row makes it possible - that's a 1/400 chance. With a short sword you'll have a 1/200 chance. Anyway that is irrelevant, because we're not talking about that 1/200 chance, we're talking about converting it into a 1/1 chance.
We're not discussing how it works. It works in the way that you cannot ever ignore a flanker. We're discussing how it should work.
If we look at how "putting down one's guard yet not be completely immobile" is handled in RAW, due to the continuity argument, there's three common effects:
AC penalty (Rage & Charge)
Becoming flat-footed (Invisible opponents & first turn of combat before initiative)
Provoking AoO (Casting a spell & Drinking a potion)
If that level 1 nobody flanking with the high level rogue were a level 1 rogue, he could find those vulnerable points just as well as the high level rogue.
Even if he knows the heart is vulnerable, it won't help him. He has to actually detect where the weak spots are, and he has to know how to hit those. He has to anticipate where the fighter's going to move, because he's not standing still.
And still, none of these are areas where you could auto-hit, I'd say the easiest place to hit from behind would be the backside of the knee, which was often protected by only leather or padding; if you cut of the knee and placed it on a shelf, someone still couldn't coup de grace it with a ranged weapon (he'd fail on a natural 1). Why should it be easier to hit it when it's on a moving combatant?
This is both a question of continuity and realism.
There is no such thing as completely protective armor in D&D. Or in the real world, I'd argue. In D&D, that would be represented as total cover.
It doesn't have to be total cover to fully hinder someone to coup de grace you.
I remember Human Weapon, a program from History channel, talking about Judo and its Samurai origins, and how warriors would throw enemies to the ground, where the weight of their armor would add to the shock of impact and momentarily stun them. After which, they'd quickly pull out a dagger or short blade, expose the enemy's neck, and slit it.
I don't know if you read that post, but that was EXACTLY the strategy I was suggesting. Trip or grapple the opponent, and either let the rogue do his work or simply hit him with a stick until he passes out. Then coup de grace would be easy. As I said, in terms of realism and balance, this is a great tactic. Considering the situation above, he'd have a pretty good chance of downing the fighter in the first turn. By the end of the second, it would be almost sure.
Once there, the relatively unskilled-at-melee archers just stabbed and prodded at any vulnerable points -- armpits, groin, eye slits, whatever. It worked rather well at killing the walking tanks.
The difference there would be that there are 10 or so half-skilled combatants (1st level warriors or the like in D&D) beating on one prone target. Yes, he would die pretty quick, regardless of armor. But there's a difference between ten warriors beating on a prone guy and some random guy trying to stab a standing, moving target with a knife. If there's ten flankers being ignored, the fighter IS going down quick.
And really, even if full plate in D&D offered perfect protection (couldn't crit someone in it), which it does not, if you weren't threatening a guy, he could do all sorts of pesky things to mess you up much worse than aid another. Can't slit your throat? Cut the straps holding the helmet to your head. Or twist it around so you can't see.
This isn't very easy to do against someone who's moving. Really, it isn't. Sure, you can, and that's nothing I object to. But it's already there, via aid another, trip, grapple, and while there's no rules about cutting the straps holding a helmet, in case it was an open helmet I'd consider that a dex-based sunder attempt. For a closed one, I'd rule you can't, because it's nearly impossible IRL. Anyways it's hard, especially as his head will be away from you. And why wouldn't this be consider "aid another"? That kind of thing is what I've always thought about when thinking of the aid action.
The main problem I have with the CDG suggestion is the auto-hit part. You should never autohit on a moving target, even if it's completely oblivious to your existance.
I'm not proposing heavy armor should make you invulnerable, I'm saying that you shouldn't be able to auto-hit a moving target in heavy armor. That's just common sense.
Take your small hammer, keep the sledge close to the ground, then swing up and smash it into the guy's groin as hard as you can. Even if the full plate completely protects, the reverberation and utter shock value will definitely leave the high level fighter open for a moment.
Again, he's not standing still, so you shouldn't auto-succeed. But sure, it should be fairly easy to hit; he's denied his dex bonus and you get +2 from flanking. Taking the characters from above, that would give the warrior a 35% chance of doing real damage. I agree that this chance is too small, but that problem stems from a whole different matter: D&D's view of armor as all-or-nothing protection rather than damage reduction. But that problem applies to all fighting in D&D (which is part of the reason I use GnGR as much as I do), so I don't think it's really relevant to the problem at hand.
And the OP never mentioned full plate, I think this whole thing is a side tangent to what was being discussed.
Well, he did, quote:
"The 19th level fighter has seen this guy hanging around before. He knows the follower is pathetic. His chance to get through the fighter's +4 Full Plate is small and his damage is pathetic even if he gets lucky and hits. "
It's my perception that he would have wanted a dexterous level 20 Fighter in studded leather to also ignore the low level "threat." Escpecially since if this is some kind of deal with the type of armor worn, it shouldn't matter if the Fighter is level 20 or level 2. Anything relating to how protected you are with full plate is an armor houserule, not a flanking houserule.
That's a very fair point though, but I think it's a non-issue, because even if you're naked:
Realistically, someone doesn't have a 100% chance to hit you if you're actively moving, even if you're not caring what he does. There's always a risk of failing when doing stressful actions, like combat, especially if you aren't a veteran at the task.
Balance-wise: If we assume that the fighter IS naked, his AC would be a pitiful 10 against the warriors 1d20+4 when attacking, and if we also gave him an AoO as I suggested he'd hit an average of 1.5 times per round, dealing an average of 7.25 (including crits) damage per round. That'd down him in less than half a minute, supposing he's unhurt, and no crits are inflicted, and the rogue doesn't hit him anything either.
Are you really so oblivious that you think the major problem with this proposed scenario is the Rogue losing his precious +2 to hit from flanking?
No, of course not. I see that the problem is the sneak attacks, but I still think it's unneeded to give free CdG's to anyone ignored. In some situations it might be worth to ignore the low threat, but is that really a problem? I mean, it's not a serious nerf to the rogue, it's rational and realistic, and it adds a little depth to the combat without introducing any new mechanics.
Yeah, so a coup de grace. You line up a shot, and automatically hit and crit. You just want to add an asterisk and say that in this case, no fort save is rolled, because you somehow think it's too powerful, but otherwise agree with my gut assessment of "the other guy can coup de grace"?
No, I don't see why. You can't coup de grace in any other situation where the person is mobile yet unable (or unwilling) to protect himself. You don't even get a coup de grace if the person is dominated by your friend.
I wouldn't have much agaist, for example, "autocrit if you hit but you still gotta hit" because it makes at least some sense, but it would be a partly new mechanic.
If realism isn't the reason to need such a houserule...then what is? It can't be game balance, you're destroying that.
To me, the change would be if my players wanted to do that, because I feel it might be rational and realistic. At least far more realistic than the options.
But anyway, I don't think it destroys game balance, since it's not going to be used often unless you usually carry around your henchman as flanker to be eaten (something I'd strongly oppose if the party was good or neutral).
Let's see...not simulationist or gamist, that leaves...narrativist, right? Is it for storytelling reasons you think a hero should be able to simply ignore an opponent? If so, has not having such rules actually ruined or restricted your plot in any game thus far?
Well, I haven't actually used any hard-written rules about it yet except that I ruled that a badger couldn't flank a purple worm, so it's purely hypothetical yet.
Of course not. My coup de grace suggestion is both realistic, and actually seems like a balanced downside to not having to worry about sneak attack from flanking. Few will ever find it worthwhile, but it might be in certain situations.
I think we have different opinions on what is realistic and not. While I am not, and I assume you are not either, professionally educated in medieval combat, I think we could leave that argument. I find auto-successes unrealistic in nearly every situation, you do not.
I partly understand your reasoning about the game balance point, but I think it doesn't have to be taken that far, so there's where I let continuity rule. I think it's enough to do what's being done in most similiar situations: Flat-footedness and AoO (though it's not common to cause both). Also, it promotes using realistic tactics: Tripping and grappling the opponent.