The problem is a perception one - that +1 bonus might not make a big difference in overall rates of success, but you will look at that number every time you roll for it. So Starting with, say, a +2 instead of a +3 is going to be obvious every time you attack.
It also means you get less feats, since you really don't want to have a +2 ability mods to hit at 6th level. This can be mitigated by spells or magic items, but most people build their characters assuming that no particular magic item will be available. If your dm definitely gives out +X weapons that might change things, but most dm's I've seen(or who post about it on any of the various fora) avoid such items as much as possible.
In other words, that number is small from everyone else's perspective, but not the player's.