Going by my tables experiences with +3s and +4s, what tends to happen is that players get better and better at 'earning' the crit rate back through their abilities, a successful saving throw agaisnt the fear spell (much less a failed save) or a successful demoralize and a flank gets you back to your normal crit rate, then throw in something like inspire courage and they might as well be a -1 creature. Your party is also likelier to crit than your crit rate would suggest because you have the action economy advantage both in terms of your regular attacks, and by utilizing reactions which don't suffer from MAP and because hero points can grant regular rerolls to missed attacks.
So you're right, and the game's tactical metagame is about reshuffling the odds.