When i am DM i look up the HD of my monsters, if they are affected.
If the PCs are lvl 1-3 and the wizzard casts scare, i know that none of my goblins has more than 6 HD and is affected by the spell.
If you ask that questions as a dc, you just have to make a difference.
Players with not much experience normally dont know the HD of the monstes. But if it´s a new spell because are able to prepare spells of a new level, you should know that there is a chance they will succeed. (the spells)
If you are an experienced, long time player you know the HD of many monsters. because you had to look for new races, etc.
But the spells normally do exactly tell you what to do.
Scare: All monsters with less than 6HD (As DM you just look if your monster has more or less than 6HD)
sleep ,deep slumber, scintillating pattern or circle of death give you a maximum (summarized) number of HD affected.
Example: Circle of Death would be able to kill 10 HD of monsters. Monsters with less HD first.
You cast them on the enemys: 3 Goblins with 1HD and 3 Orcs with 2 HD
and 1 Ogre with 3HD
The spell affects monsters with the smallest number of HD first.
Now you just have to summarize.
3*1HD + 3*2HD = 9HD The Ogre, because he has 3 Hit dice would be affected as the third group. But 9HD + 3HD = 12 HD That would exceed the maximum summarized number. So, the Ogre is not affected by the spell.
Hope this helps you, but the short answer already gave Runestar. The DM has to look up the Hit Dice and the Player, if he doesnt know the Monster´s HD, just has to cross his fingers that his spell will work ^^