Yeah, what MoogleEmpMog said.
It's not so much doing the adding as remembering the rules for each thing that adds (or subtracts) from a die roll.
Players aren't just doing "10 + 2 - 1 + 1 + 2 = 14". They're doing "Okay, My Attack bonus is 10, plus I charged, that's +2. I'm on the steps below McSneery. Is that a minus one? Or does he just get the +1 for higher ground? Ah, I'll just take the minus one. I have AlmostBannedSpell and CoolNewSpell on me for another +3. Wait, do they stack? (Flip, flip, flip) Yeah, okay, one's a Dumb Luck bonus and the other is Chutzpah, so they do. Okay, so what's that so far? 15? Errr, 14. Right. Wait, did I count charging already?"
You have to remember the rules surrounding each bonus, plus remember to remember them all. That is where having too many bonus types get to be an issue, possibly slowing a game down considerably, and one's ability to do math is only slightly related to the problem.