"Being better with axes than with any other weapon group" means you only ever use axes.
If you're only using axes, your axe attacks should be on par with other fighters' attacks (with any type of weapon).
If your axes are just as good as their weapons, and your non-axes are worse than your axes, then axe specialization necessarily means that your axes are on the baseline, and your non-axes are worse than the baseline.
Ergo, axe specialization serves only to punish the specialist for using non-axes.
Or because you are focused on axes, your axes are a little better than par with other fighters, but your non axes are worse than baseline.
Here's my viewpoint with an example. Scale of 1-5, 5 best.
Generic fighter (all weapons 4)
Specialist fighter (spec 5, all others 3)
Clerics, magic users everyone else etc (weapons 1,2, or 3)
So the specialist isnt gimped, he's still better than other classes at fighting.
You trade versatility for specif effectiveness.
I know, I know, "but in order to be all they can be now all fighters have to be specialists!!!!!!!!!!"
I say "why" and "bull".
Does your character really have to be maxed out and top to hit to be effective? I don't get the part where people explain the fighter is gimped if he has the opportunity to specialize and doesn't.
And you would still plan adventures on the baseline fighter. The specialist would cut through (pun intended) some encounters like butter, but not all of them.
I'm rambling and not writing coherent thought passages, I know. Its a tough subject.
Bottomline? If the axe specialist finds a flaming longsword of doom, either the opportunities to set things on fire offset his spec loss to his character or they don't, he made the choice to specialize, its supposed to be limiting.