Strongly disagree here.
First, I don't find that caster turns typically take longer than martial character turns, unless they are casting a complicated spell at higher levels. Most of the time, their turn involves an attack roll, plus some damage dice, or a saving through and an effect. But on the other hand, martial characters also start doing a bunch of extra stuff on their turns at higher levels.
And mastery adds significant time to my games, especially when working with new players or unfamiliar players, which is exactly what the OP describes. Maybe you are playing only with veterans who know the rule inside out, but at my games masteries consistently add confusion, players are forever forgetting to use them and then trying to backtrack, and the extra rolls alone add more time. The flow chart of their turn is much more complicated.
As for more fun, I'm going to give my players TwoSix's option of taking a flat +2 to strength or dexterity instead, and I bet most of them take it.
So for my games, your statement is just flat wrong. With weapon masteries, those player turns typically take much longer than for other players. It's by far my most disliked element of 5.5e.