Rolling a 20 should always be the best possible result...
I dont get this line of arguement. A 20 is just a crit when you are dealing damage. A crit knows no morality. What a mess it would be if the crit had to decide what was "best" for the player characters before it took effect. PC: "Ha, a 20! I crit the Evil Wizard and deal 27 points!" DM: "No! In a sudden surprise to you, your critical actually deals healing to the Wizard. It is revealed at this point that the Wizard is not Evil at all and you had been mislead by his Vizier! BTW, the Vizier screams in pain back in the city and takes 27 points!" PC: "Huh?"
Anyway, I have a lot of sympathy for the DMs ruling in this case. I think if the players had felt that the DM screwed them at the time, they would have pressed their case then and there. PC: "I was counting on the fact that subdual damage couldnt kill? Rules check?" or PC: "I thought a 20 could only bring me good things?" But, frankly, the result is pretty reasonable; I would certainly accept it.
As player or DM, I would want the rules to cover the possibility that an Evil PC might accidentally beat his prisoner to death while trying to shut her up. The DMs job is to referee things like this, and part of this is being able to make up sensible rules and results on the spot. If its later found that the DMs on-the-spot decision conflicts with the 3E rules, well; things like this happen. In this particular case, I would expect the DM to be able to go back to a reasonable playgroup and say, "Hey, I was looking at the subdual rules and they say you cant kill someone while trying to subdue them. I think thats lame. I'm Rule Zeroing it to work the way it did last week. Any comments?"