Because there's only so much page count and design effort to put in at one time. According to a quick google search, there are 58 species allowed in Adventurer's League Forgotten Realms campaigns (which is more than I thought, I would have guessed 40), and 183 species total across all books. That's way too much to include in the core book, especially when a lot of those are extremely obscure or are very setting specific (like the Eberron version of elves).
I'm a big fan of variety, but trying to cram 183 species either requires too much of the 384 pages in the PHB or ends up making them mechanically indistinct. It's also going to be absolutely overwhelming for a new player to get a handle on, that's way more options than needed to hit analysis paralysis.