First off, I don't think it'll break anything, so if it seems worthwhile, go ahead.
Personally though, I think the classes that are supposed to be extra-effective unarmored (barbarian and monk) already are, so I wouldn't start adding in extra nickel and dime bonuses.