As long as we're talking 'specialist wizard' they will suffer - by definition we're using a standard wizard (whatever form that takes in 5e) and making something that cannot exceed and - likely I think - won't equal that standard. 2e and various iterations of the concept in 3.x (Pathfinder and 3.0 primarily, as those are what I have experience with in this regard) both exemplify this. My favorite version of the illusionist, in my opinion the most flavorful, was the 1e version - which is literally impossible to make in any later edition because of certain 'spell schools' being denied them.
For specialist wizards to really work, I think a couple of things need to happen. Firstly, lose the spell schools that have existed to this point. Or at least, change them around. Reorganize the spell schools into a smaller number (say, five or six) and organize them around thematic lines, as opposed to .... whatever they're organized around now. Each of these new schools would, I think, suggest a certain specialist class.
Say, Necromancer, Illusionist, Summoning, Enchantment (unless that will step on any extant psi class too much), Seer, and maybe Abjurer (or whatever you want to name the anti-magic school). Note that I didn't include Evoker. A big complaint about specialist mages is that they become useless in combat - well then, break up the combat magics into different themes (not six identical spells but reflavor some of the extant ones) and then everyone can take them. (You could still make an evoker, I suppose, make it an optional add on school wise).
Anyway, from there, you build the specialist class as it's own class based but not exclusive to that school. This way you don't have specialist wizards that suck at doing anything OTHER than their specialty *and* you dont' have necromancers that are, for example, strangely really good at healing. This does create a problem in that you now need seperate spell lists for everyone which might be a deal breaker for many.