Klaus said:
I use a d20 variant of the Buffy RPG mechanic.
A stake is essentially a "piercing" club (medium damage 1d6, crit 20/x2).
If you score a critical hit with your stake, calculate the regular damage for the attack (without the crit) and for the crit (x2). If the critical hit damage is enough to bring the vampire to 0 hp or lower, the stake has pierced the heart and the vamp goes POOF! If not, then the vampire only suffers the regular damage. A kind DM can rule that stakes bypass a vampire's DR (I do).
Same goes for beheading. If you do enough damage with a slashing weapon's critical hit to bring the vampire's hp to 0 or lower, you have severed its head and it dies instantly. If not, the vamp suffers only the weapon's regular damage.
This reflects the role of hp as the "ability to turn deadly blows into near misses" and such. So, like in Buffy, you'll want to pound that vampire for a while before trying to stake or behead it.
A stake seems more like an improvised weapon to me; -4 to attacks, and 1d4 damage, like a dagger. It doesn't feel right for people to use it as a regular melee weapon. It seems intrinsically unsuitable for sword fights (or even knife fights), and IMHO even an exotic weapon proficiency wouldn't help.
Maybe I'm a harsh DM, but I wouldn't let a stake bypass DR unless it makes the killing blow. From the Buffy series it seems that if the stake misses the heart it is doesn't hurt the vampire at all; the only time she uses the stake is to finish a vampire off. In D20 I'd say that if you are carrying a stake and make a melee attack that reduces a vampire to 0 hit points, then instead of going to gaseous form it is "dusted."
As a small improvised weapon a stake won't do much damage (1d4, like a dagger). The presumption is that you'd roll your attacks and damages with your primary attack, but would keep alert for the possibility of a heart shot; if you reduce the vampire to 0 hp then you can automatically follow with a killing blow with the stake. This is a different mechanic than critical hits and called shots; it is flavor for the mechanical effect of preventing a vampire's gaseous form defense. Since a stake uses up a hand it prevents you from using a shield or wielding a weapon two-handed; this is the mechanical balance to the advantage of preventing the assumption of gaseous form.
Now according to this logic a character who is carrying a slashing weapon (like a sword) should be able to do the same thing; cut off the vampire's head when it reduces to 0 hit points, thereby preventing it from going gaseous. I would allow this if the slashing weapon was being held back in readiness for the head shot. But I wouldn't allow it if the sword was being used as the primary atack. It seems balanced mechanically to let a stake-wielding character bypass the gaseous form defense of a vampire because it is an inconvenience to have to carry an otherwise useless stake in one hand. But characters are not inconvenienced by having to carry a sword!
But I like Klaus's basic idea about criticals, except I would like to point out that undead don't take extra damage from critical hits. But if a character with a slashing weapon is fighting a vampire let him try to confirm a critical threat; if the vampire is reduced to 0 hit points with a critical from a slashing weapon, it is a decapitation, which prevents the vampire from going gaseous. There would be no extra damage from the critical; it just prevents the vampire from going gaseous; the vampire is destroyed instead.
Incidentally, isn't this how weapons with special properties on criticals function against undead? You roll to see if the critical is confirmed, and if it is the flaming burst (or whatever) activates, even though the base weapon damage is not multiplied. In other words, undead *aren't* immune to criticals; it is just that the critical multiplier of all attacks are reduced to 1 against undead.
Anyways, I would say there are two cases when a vampire can be destroyed instead of made to go gaseous; if a slashing weapon or a stake is held "in reserve" until the character's primary attack reduces the vampire to 0 hit points, and if a critical with a slashing weapon reduced the vampire to 0 hit points.