My 19th level game has a fighter a ranger and a paladin, all with various weapon masteries. They LOVE the weapon mastery add on to their characters (I let them switch to 5.5 versions).
And the game hasn't shown any kind of lag or slow down in play at all.
I have a very different experience in the 7 1-20 or 3-20 5.5E campaigns I played.
The gaming groups I play with are mixed when it comes to masteries. The ones that like them typically wanted a bigger change from with the new rules. They like the masteries, but still dislike the rules generally. Those that dislike or don't care about the masteries are generally happier with the new rules in general (and were happy with 5E before they came out).
One thing that is consistent though is a combat turn takes a LOT longer for martials in 5.5E, easily twice as long as it did in 5E. Time for casters is actually less than it was in 5E
And high level martials taking longer to act than high level casters? That hasn't been my experience at all. Weapon masteries, for example, are a quick, easy add on effect.
Like I said, I have played a lot of high level 5.5E and that is not my experience.
Masteries objectively cause many more rolls and those that don't explicitly cause more rolls, cause extra actions or extra math. Topple is literally twice as many rolls on the turn of the PC that has it, and then is more rolls for your allies targeting the bad guy too (whether using ranged or melee). Vex is more rolls. Sap is one more roll for every guy you hit. Push is used for things like pushing enemies into spirit guardians and then there is a save and a damage roll for that. Nick lets you attack without a bonus action meaning you can do that and use a bonus action where without it you could only do one of those. Cleave is an extra roll when you hit with another enemy in reach and although Graze does not cause extra rolls it does cause extra damage (and the time it takes to manage hit points). The only mastery that does not objectively increase the time it takes in combat is Slow.
Then after masteries you have things like Heroic Warrior, getting extra movement every time you use Second Wind (in addition to using it more often), Monks doing more attacks, Rogues throwing save effects on their sneak attacks (in addition to making sneak attacks with a Truestrike bonus action) ....
Non-caster Martials are FAR more capable in 5.5E than they were in 5E and most of that additional capability is not from damage, it comes from being able to do more things in combat, both through weapon masteries and through other "things" that take time.
Whereas unprepared players of high level casters can take ages to not only decide on a given spell but then take a while to sort out all the effects, placement etc.
I don't play at high level with unprepared players and anyone who plays a 1-20 campaign should be pretty experienced by the end of it as we are talking about well over 40 hours of play.
Spell casters were sped up quite a bit in 5.5E by how they simplified spells. 5.5 spells are much more straightforward with little variation or rules complications. For example, Tashas Laughter and Command now affect everything, no more checking Intelligence or if a creature knows a Language. There are no more spells that "summon 8 allies that all get their own turn" and the spells that still do summon one ally use a specific bespoke stat block, so there is no searching the Monster Manual for a stat block.