It's like changing the difficulty level from Medium to Hard, I say it depends on what kind of challenge you're looking for.
And you'll at least have some magic armor/weapons, so you're probably not going to be a full 6 points behind.
All in all, everyone has plays the game a little different...