There are two aspects to being a Rat Bastard DM. A plot aspect and a tactical aspect. Both are important. But the basic philosophy for both is the same. You have to let the players get themselves into a situation that gets much worse than they expected, and you have to do so in a plausible manner.
The tactical side is easy. Anyone can put together a decently challenging fight, just pick the right monster and run it. But being a tactically capable rat bastard requires more than just picking the right monster, and inflicting damage. You have to be able to run a fight where the players get the sense that they cannot win, despite it being a completely fair fight. That means no min maxing the tactics to the players in advance, and no pulling things out of your arse just to make the fight harder if it turns out to be a bit too easy. What it does mean is using very smart tactics. Here is an example:
- If you have a villain with Improved Disarm, than have some of the cannon fodder delay their action until after the villain. If a Disarm succeeds, have the fodder type pick up the disarmed weapon and then run. This qualifies as being somewhat Rat Bastardly because instead of just being mildly inconvenient, it is suddenly very dangerous. If you had a Fighter who had heavily specialized in the weapon, and it was say, a +2 sword, you just inflicted a net -3 to hit and -5 to damage since that fighter can no longer get the benefit of the weapon enchantments or his Weapon specific feats. It is also fair simply because it just a good idea, and unlike sunder, the player still has a chance to get the weapon back. Bonus points if you hand the weapon to the villain and he drops the player with his own weapon.
- Using Bull Rush with a troll at the top of a cliff is a pretty good RBDM tactic. The troll will regenerate from the falling damage. The player will not. The troll has a huge bonus to the Bull rush due to size and a very nice strength bonus. The player is now also probably on his own against a troll with no help nearby.
- Min Maxing your tactics to your players is not a good plan. Unless you can justify it fairly. When using organized opposition, I will often make a point of having some opponents flee. It lets you skip the mop up part of combat, and the players still win. But if they let enough opponents survive and report back, the survivors get to report to their superiors what the players look like, and what they did to wipe out their opponents. Then when you pull an ambush that starts with dropping a Silence spell next to the casters and nailing the fighters with some spells calling for Will Saves, it is not cheating.
Plot wise it is a bit harder to pull off, but not by a whole lot. The trick to being a good Rat Bastard DM in this regard are the following.
- Always give the players a choice. This is the most important thing to do. But I am not talking about a "Keep the money or return the money" kind of choice, where the 'right' choice is obvious. You need to present your players with choices where all presented options simply suck. So what I am talking about here is a "Help an old enemy escape or allow a new and dangerous villain to succeed in his plan" kind of choice. Or "Protect the corrupt king or kill him and become fugitives". If they can find an actual good option, let them run with it though.
- Give the players enough rope to hang themselves. Once the players make a choice that can harm them, then use it against them at some point in the near future. It can be an option you prepared, or just a stunningly bad idea they came up with all on their own.
- Not exactly required, but a mastery of the Xanatos Gambit can't hurt either. [LINK]http://tvtropes.org/pmwiki/pmwiki.php/Main/XanatosGambit[/LINK]
One last thing though. You absolutely MUST be willing to let your players lose. Do not get DM vs player about it, but your Villains and even your monsters must be played as though they are trying to win. In combat, this means you should not fudge rolls for the players to let them survive a close combat. This means willing to run fights the players must run away from, and while you must make it obvious and give them that chance, you must also be willing to pull a TPK to make the point. This also applies to plot. If the players screw up, then they should have to earn 2nd chances to make it right, and you must be prepared to carry things out to the logical conclusion if they fail.
Being an effective RBDM is difficult, because you must run right up to the line between providing entertaining and challenging adversaries, and being a cruel and petty ass. They say that a Perfect Crime is when someone does something that they should be arrested for, but they manage to do it without breaking any laws. Being an RBDM is very much like that. You have to do the meanest and cruelest things you can to your players without giving them any reason to be pissed off at you personally. If you can pull it off, your players will be grateful, though if you are doing it right, they wont exactly thank you.
END COMMUNICATION