Nah, I'm the DM and this happened last night:
DM: Wraith attacks, ouch a 20, confirms. That's gonna hurt with a d6 CON drain.
Player: Well, it can't kill me at least.
DM: Hmm, that sounds like a good way to curse yourself..
Die Roll: 6
Die Roll: 6
DM: Yup, that was a self-curse alrighty (glad I roll in the open).
The player, and his two remaining CON points survived the fight after a brief bout of incapacitation. It's a long way to the nearest Restoration spell so I wanted to be sure about the damage inflicted.
So for things that drain energy levels, the drain is doubled as well?
(ie. a crit from a spectre drains FOUR levels?!)