That's a great way to totally nerf the spell. If that's your intention, go for it. However, the single reason Sleep is a good spell and not a total pile of crap is that if you can hit and they fail their save, they are completely helpless for one or more rounds.