Well, it would make the game grim'n'gritty, that's sure. It would also, however, make it a lot simpler, and not only in the good way. No character progression is just as unrealistic as 17th level fighters. As is the idea that characters cannot improve, ever - and I'm not talking about combat, either. If my character decides to take up blacksmithing, I expect him to eventually get good at it.
A very slow advancement could work better. You could give XP in such a way that most characters will reach 5th level by the time a normal campaign has reached 20th.
This, however, introduces the problem of insufficient granularity: it makes the level jump a much higher cliff. Having a character suddenly double in power feels weird.
To solve that, you could split each level into several steps. Every character gets one HP every now and then, one point of BAB, one point of one saving throw, one skill point, one special ability (maybe after an appropriate quest), until they get the full level. It would take a bit of tweaking, but it could work.
If you let the PCs choose what to get, you've basically got point-based D&D.