I would say the best lore is in the AD&D era. There were some interesting things in 3.x, but a lot of it was pretty much a rehash of what had come before. 4e attempted to do very new things, but at the expense of the old lore, so naturally that annoyed some people (incoming Forgotten Realms rant, be warned).
5e has pretty weak lore, I haven't run across much I find interesting, and I continue to mine old lore because hey, they can't take it away from me, lol. The Rain of Colorless Fire or the Invoked Devastation is way more interesting to reference than "Lolth failed to take over the Realms, Shar failed to take over the Realms, now Tiamat fails to take over the Realms...hey kids, it's Acererak! Remember him? Yeah, he's going to fail too, but at least his plan has balls- he's going to try and make his own god that he can control!".
Forgotten Realms and 4e. Where to begin. The Year of Blue Fire was a hot mess. Nobody can argue that it wasn't. Attempting to radically change the status quo of the Realms is a valiant effort though, but it was never going to fly- the lore and the characters are what makes the setting popular. And in the end, nothing changed, either people were put in carbonite for a 100 years and came back, or they just retconned it anyways.
Because the Realms needs it's status quo changed. It really does. But like Batman, who really should turn over the mantle to any one of his successors, he's too popular, people want "their" Batman.
Just last night, a player asked me for some perspective. "You keep saying the Forgotten Realms is full of powerful people. So what if, uh, Arthas Menethil from Warcraft, full power Lich King, all his armies of undead, invade the Realms?"
"Oh man", says I. "You have no idea. I could go on about the current movers and shakers and levels and feats and all that, but you know what, let me introduce you to one guy. His name is Larloch. He lived in an era where archmages were tossing around 11th level spells (there was a 12th but that didn't go so well). He made a super powerful artifact, a ton of unique spells, and he made himself a Lich for funsies. He's been around for 2000 years since then. He has a huge city of undead, a vast horde under his control, a veritable army of undead wizards, and you know what? He really doesn't give a crap. His goals are advancing his own magical power not for world domination, he could do that, but out of the sheer thrill of it. It's the only thing he cares about. Any serious thread would be ROFLstomped; his bio says he's spent centuries working on contingency plans and has tactics for any situation, and if he doesn't want to fight at the moment, he just...leaves. He's a literal Batman wizard, if Batman was over 2000 years old and had every known spell and some he made up for the hell of it. And this is just one guy."
"Woah. Is he like, the end boss?"
"Haven't you been listening? No! He's a random NPC that has no real goals beyond "I want to see how strong I can get". Ssazz Tam, one of the most frightening dudes around, is way weaker than he is. He stole that super artifact, the Death Moon Orb, and Larloch was basically "oh, that old thing? You want to take over the world with it? Have at it, I don't care."!
I love the Forgotten Realms lore, but as a setting? One where the actions of a ragtag bunch of murderhobos really matter, when you can't throw a rock without hitting a guy with double digit levels and a backstory that could fill a novel (at least)? Kind of ridiculous.
And yet, for all the magic items, the ancient civilizations, the super powerful casters, is there a magical industrial revolution? Nope. It's been in this kinda sorta early Rennaisance since the 80's. It' won't change. Oh there's a couple magocracies, but the Thayans are basically the Sith, and the Halruaans nobody talks about. The most advanced thing they have is an arquebus (unless you count some wacky inventions by their Leonardo DaVinci expy). Well, ok, they had wheel lock pistols, but those came from outer space. No really, it's canon! But don't worry, the Gods don't like guns, so almost nobody makes "smoke powder" anymore.
Eberron gets more done as a setting with half the levels. I'm not saying the Realms should be Eberron, mind you. But nothing really changes, man. Every decade or so there's a big EVENT like the not-Mongols invading, a massive flight of dragons, the Gods having some soap opera drama, but it doesn't matter, it'll settle down and things will get back to normal in a year or so.