I just went back through the PHB and found a couple of interesting things: Under the Rogue entry it states that you get to use you SA damage any time that an opponent cannot effectively defend himself from your attack (I think lying Prone with a -4 to your AC fits this criteria). It also states that the Rogue's SA deals extra damage any time your target would be denied "A" Dex bonus (it doesn't specify that the opponent needs to be denied it's ENTIRE Dex bonus). And under Helpless Defenders, it states that the Rogue may use his SA damage, and they take a -4 penalty to AC. And then under Feint, you get to use your SA when all you're doing is misleading an opponent to he can't effectively dodge your next blow. So my reasoning is this: how can you SA someone in a Flank or after a Feint when the opponent is still on his feet, but you cannot when they are lying flat on the ground?
You think falling prone means you can't effectively defend yourself? My monk got knocked on his arse 2 weeks ago, and he still dodged some attacks and even
killed 2 Large Skum while lying prone, because it wasn't worth the bother of standing back up first. Think breakdancing or capoiera....
The reason you can SA when flanking is that the opponent's attention is too divided between someone directly behind them and someone directly in front of them; as soon as the opponent tries to glance or turn to see the opponent behind them, your rogue stabs them immediately while he's not looking; as soon as the opponent turns around to face your flanking ally, your rogue stabs them again while they're not looking. And when the opponent is flat-footed, they're just standing there, unable to move; paralyzed with fear, or surprise, or whatever; you got the jump on them, and make the first strike as they're still trying to figure out what to do.
Merely being inconvenienced by falling down and needing to roll, instead of jump, away from an attack does not make you helpless or unable to defend yourself. You're still mobile and you're still fully aware. If you still have a chance to avoid the attack, and can still act (like taking an attack of opportunity), there's no way you can tell me you're 'unable to effectively defend yourself.' You can still use your sword and shield or whatever, and you can still fight and maim and kill enemies, so how do you figure you're unable to effectively defend yourself? You can still avoid getting hit, and you can still kill your opponents. That's not 'effective'? Please.
Nowhere in the rules does it call prone or other AC penalties a "Dex penalty" or "denying them X amount of Dex bonus". That's becasue they're not. They're situations that limit or hinder your mobility but not fully, and just make it harder to avoid attacks, not impossible. Taking a penalty doesn't "deny" you any Dexterity bonus, either. You still get your full bonus, you just get some penalty added in. This is basic math. 10+4-4 is 10; just because some wolf tripped you doesn't mean you lost 4 points of your fighter's Dexterity bonus to AC. You still have that +4. You just happen to also have a miscellaneous -4 applied to your total AC. You did NOT just suddenly become as inflexible, ungraceful, uncoordinated, and slow of reflex as a drunken old man. You're still a fighter with incredible Dexterity, who's just in a disadvantageous position for the moment.
It's no different from a fighter with 10 Dexterity fighting normally with no penalties; he's got the same AC, he's just not getting any bonuses or penalties to it. What about if that fighter has full plate armor and a tower shield? 22 AC? What's happening when he gets knocked prone for just 18 AC? Or what about if that fighter was suffering from a nasty Dex poison, to where his Dexterity is reduced to 1 currently, making his AC 17 normally, or 13 when prone? Tell me how the barely-mobile-already fighter is being denied 4 points of Dex AC when he was just a standing pile of armor in the first place. He already had as little Dex as possible without being paralyzed. Yet technically, he was not vulnerable to Sneak Attacks while standing around, despite his 1 Dex. Despite being all but paralyzed, he was still able to move a little bit, awkwardly and slowly, and yet was still able to avoid being Sneak Attacked.
You only need to budge an inch for a knife to graze you rather than sticking in your throat, or kidney, or whatever the rogue was aiming for. You can't do that when you're utterly immobile, or completely unaware that someone's snuck up behind you and is about to ram 6 inches of steel through your back, or too busy trying to avoid the other guy who's trying to chop your head off while the rogue gets behind you and stabs while you're ducking under the other guy's sword. Or when the rogue bluffs and makes you look away for a moment, making you think that his roguish buddy has just snuck behind you and is about to stab you in the back, with the rogue himself actually stabbing you in the kidney while your attention is diverted away from his knife.