So let's look at what's really going on with critical hits. Player side, most people do negligible amounts of extra damage, as you've noted. In 3e, the basic crit was equal to 2 regular attacks, so if you crit, you were up one attack. Weapons that gave you more than a 5% extra chance to crit did less base damage (meaningless if you have high static damage), weapons that didn't improve crit rate did more damage (so rare outside of edge cases like murdering a sleeping foe that all that extra damage is likely going to be wasted on a mook or a guy fuming on hit points anyways). From a design perspective, enough things were outright immune to crits in this edition (and you could always slap Fortification armor on your BBEG) that this was bread and circuses; big numbers make ape brain happy.
And on the DM side? Most crits were 20/x2 for monsters, so it remained a 5% chance for an extra hit. Sometimes that was a very scary extra hit, but it was acceptable.
4e came along and tried to simplify the process by turning crits into max damage. Well that was easy, but players stopped feeling super excited about crits in of themselves, unless you were chucking out a high damage move; many players were happy for the auto hit, especially when using a power that completely wrecked an encounter in other ways ("yes, nat 20, the BBEG is dominated/stunned for 1 turn, let's clean house, boys!").
They had ways to do more critical damage, and the people who liked that sort of thing gravitated towards it, even though the only way to "crit fish" and have this be a truly relevant moment were...uh, attack more times, or be an Avenger and ROLL TWICE, MAN! Woo!
Now in 5e, everyone goes on about Rogues and Paladins, but the class that has the easiest time getting a high crit range and makes the most attacks? At best they're getting another 6.5-7 damage out of a crit. Wow. A crit isn't even worth a whole extra attack! The classes that have huge damage pools? They can roll twice with advantage. Maybe super advantage with Luck or Elven Accuracy.
And all of this pales compared to monsters. See, while most players only have to look forward to another die (or two), many monsters are tossing out multiple dice of damage, because WotC decided that, for the most part, monster damage is a function of their Challenge Rating, not what attack they're really using. Oh they make some attempts, but often "this monster does an extra die of damage because we gave it a special ability (Gladiator)" or "we just made this number up" (Giant Ape- where is the logic behind a 7d6 rock? How big a rock is it? How many of these rocks does it have? Don't worry about it!).
The DM is always going to get more crits than any single player; he rolls tons of dice for a seemingly infinite amount of monsters. And after awhile, when he does, he's getting way more out of it, because sure it's fair that that random CR 7 crits for 55 damage, while you crit for 19, why wouldn't it be?
And WotC knows this is borked, they tried to address that in the "New and Improved Flavor 5e" playtest. And the people said "nooo, we need that 5% chance to bring a guy down to 0 hit points so he can be propped up by a few 1st level spell slots! That's the only way the game is challenging!".
So here we are. Crits can simply not matter for players, but be back breaking for DM's. Anything you do to make crits better for players makes crits better for monsters, so the game becomes more swingy. Some people will like that, others will be ok as long as they do moar damage.
Others will be hit by a random crit no matter how careful they are and go from full hit points to zero and it's ok, because I can heal you for 7.5 damage as a bonus action and you can keep playing! Isn't that awesome?
