One downside is that low-level characters don't have as many different options in each encounter, which could a) make things get a bit boring and b) tend towards shorter adventuring days then necessary. If I were going to restrict the game to three levels, I would consider giving out more powers...