Lore is meant to inspire. For monsters, it’s meant to inspire combat encounters, NPCs, and adventures you could include in your campaign. For player options, it’s meant to inspire what you could play and how you could roleplay your character. WotC chose to trade some lore in the monster section for more art and monster stat blocks.
And I don’t think the lore in the original 5e rulebooks was very interesting or inspiring. And most of it is unnecessary. Everyone knows what Elves, Dwarves, Hobbits, and Humans are. Even for the more exotic races, people know what Dragons, Devils, Angels, and Giants are. Do we need over 2 pages of lore per race that just reinforces traditional tropes about Dwarves if most people already know what Tolkien-esque Dwarves are like? I don’t think we do. It’s better to just have a few short paragraphs giving the basics for the players and DM to build off of.
Not to mention that the original PHB wasted a page on a ton of Human names from the Forgotten Realms. That’s not lore that inspires people to make adventures or characters. I’m glad that was cut.
But for monsters, the lore should be harmony with the mechanics and written around their intended function in campaigns. Mind Flayers reproduce via implanting tadpoles in the brains of captured victims, so they should have abilities that allow for them to non-lethally neutralize enemies (stun, paralyze, mind control). Death Knights command armies of undead, so they should have abilities that allow them to control undead and buff their minions. Giants are big, prideful human-like monsters that throw rocks and squash people. You don’t need to have multiple pages of lore explaining their culture, just a general idea of how their mechanics are based on and inform their behaviors as creatures.
Even though there’s roughly the same amount of lore for the Empyreans, I can say the new lore inspires me more than the original 2014 lore ever did. The inclusion of Iotas and change in art made me go from completely uninterested in Empyreans to brainstorming how I could integrate this with the world I’m currently working on. For me, the trade off of more art and monsters is overall worth it. A picture is worth a thousand words, and alternate versions of monsters can often inspire in ways that paragraphs of lore can’t. And sometimes less is more, and you should take the time to write a shorter letter. I think the 2024 Dwarven lore section is as useful at informing players what Dwarves are about as the 2014 section, it’s just more efficient and doesn’t take 2 pages to say “yep, they’re Dwarves, like Gimli, you know what they’re like.”
Also, you can find basically all the D&D lore you need for free in wikis and YouTube videos. And WotC will presumably keep making the Fizban’s/Bigby’s type books for other creature types to expand upon lore for the more interesting/iconic monsters.