Yeah, unfortunately I think it's a bad idea. Not for the fact that a player will have a 20 or an 18, as you can adjust encounters for that if all characters had them. The problem is that the standard rolling can make a large gap between characters. A monster that a level 1 with a 20 can handle...