Apologies for breaking this up piecemeal, but, you have a few different issues here.
Honest questions.
1. Does it have to be all or nothing? Couldn't a niche subject be included in an otherwise generically useful book?
Of course. That's generally what's being done. We get new monsters, classes, rules in modules right now. Ghosts of Saltmarsh includes naval rules for example.
2. Did more players want a fully fleshed out Firbolg or Tabaxi or demonic vehicle battles than a psion? If not why create them since niche isn't the way to take 5e?
Well, there is the case of scale here. A single page for a Firbolg or Tabaxi using existing mechanics is hardly the same as a 300 page rulebook for psionics.
3. If 5e had such limited development resources at it's disposal then how does Rick and Morty adventures, Acquisitions Inc subsystems, and Stranger Things box sets expand the game for the Day 1 player?
Note, none of those are developed in house and all of them are licensed out. I think. I could be wrong there. But, like the early 5e modules, the work isn't being done in house.
4. As an extension of 3, what percentage of official books in 5e are "generically useful" meaning that most of the included content can fit into any campaign. By my count that is 6 books (the big 3, the two extra monster books, and Xanathars). Aren't non-generically-useful books more niche than those that are? Why are they so rare in the release schedule?
What other non-generically useful books? After the core 3, Volo's, Xanathar's and ...sorry,what other monster book? what is there? Modules AFAIK. That's it. SCAG I suppose. But, even that is directly tied to five or six other products.
It basically boils down to this. If it's not going to sell in the 100k books range, WotC isn't interested in doing it. It's that simple. Is a psionic book going to sell like that? I dunno, but, I doubt it since none ever have. Now, if they do a Darksun book, similar to the Eberron book, then, sure, that's where the psionics is going to be found. I can totally see that. But, that's also going to appeal to much broader segment of the audience than just psionic fans.