I certainly dont want to get into this too deeply, but if you follow the setting from RT, through 3rd to 7th (and beyond) you are calling out what is essentially the formative period of the current and now established view of what 40K is.
No.
Sorry. I get that if that's when you started you might think that, but the difference between 2nd edition and 3rd in terms of concepts is nearly non-existent. The main difference is tone and emphasis - and note grimdark vs not, rather humourlessness and starry-eye'd-ness towards the Imperium rather than the cheer and cynicism of 2nd, which was even more pronounced in 1st, as you've acknowledged. Almost everything that really matters in 40K, except for the later-added races, was in place in 2nd edition. That's where the consistency you're discussing comes from to the limited extent it exists. It's also worth noting that the three races added in 3rd have been pretty inconsistent in how they've been portrayed - more so than the rest of the setting even.
Its an art style and tone that absolutely had a reset in 3rd, that carried on for over a quarter of a century.
Not that long. That style only lasted until about 2013, maybe a little later in some products, and then slowly started dying, perhaps because they were acting as a hard-limiter on 40K's appeal. Modern 40K is far closer to 2nd edition in most ways than it is to say, 5th, setting and tone-wise. The art style isn't markedly different from 2nd to 3rd and onwards, either - that's a wild and weird thing to say in the face of the fact that many of the models in 40K lasted from 2nd edition until in many cases very recently! Eldar had models which were 30 years old! Is the modern Eldar aesthetic distinguishable from the 1992/1993 one? No. It is not. The same is largely true for everyone except the Marines, who military-fetish'd up a fair bit. But no-one is expecting the Beakies to come back, even though in a better world they would!
The rule books, codex books, white dwarf, Black Library novels, Horus Heresy series, the Forge World Black Books, from 3rd, to today? There is a very very consistent portrayal
Nah. There's an obvious change in the rule books/codices around 7th, and unarguably 8th. Suddenly they start being much more interested in the universe as a whole, the level of grimdark goes from like 11 to about 8 (esp. as they stop insisting the entire galaxy will definitely be eaten by tyranids, no question about it), so still very grimdark, but we're back at 2nd edition levels of grimdark. The Space Marines aside amount of gothic-ness starts coming up again, whereas from 3rd through parts of 7th, it had been going down, replaced instead by militarism/military-fetish stuff (which instead gets silo'd off into the Marines).
As for "consistent" re: anything about the Black Library or codices, not really. One BL book will have a completely different vision to another, even between the better authors. The codices frequently contradict themselves from edition to edition, and sometimes even within a codex there will be non-in-character contradictions because the writers aren't paying attention - this is lot less bad now than it was 4th/5th/6th, note. But huge revisions and retcons, some unintentional, happened all the time. In more recent years there's been more intentionality to it.
As I noted earlier, there is a limited kind of consistency in terms of what's going on and how things work, but that is almost all rooted in 2nd edition precepts, not 3rd and later. Especially not 4th/5th/6th, which were the nadir of lore for 40K. If there's a big idea about 40K that doesn't relate to newer xenos, you can bet it got locked down in the 2nd edition and original Epic Scale era.
It's very easy to tell this if you've been involved with 40K as long as I have (since 1988). Listen to a podcast about it and people who only started in 2015 will be quoting lore concepts that are from 2nd edition or earlier constantly - and yeah they often think such and such latter-day book introduced something, until they have some lore boffin correct them. They're the building blocks of the setting.
Easter eggs of the old take on the setting may exist, but what you are calling out (3rd to 7th) + HH, is the foundation of 40K to the vast vast majority of its fans.
They're confusing 2nd edition and Epic Scale with those editions, if they think that. 1st (Rogue Trader) is rather different (and smarter but less ornate and coherent) but that's a longer discussion. 8th/9th/10th have distinctly revised 40K into something that's a lot closer to 2nd, hugely so.
Otherwise, I pretty much dread having the wider culture look to hard at 40K, I dont want to see what it gets twisted into.
I don't think it'll get twisted at all, that's the thing, and I think that, for various reasons, is the actual issue. There are different groups of people who worry:
1) People who like some Imperium faction(s) or other and are afraid that said factions(s) being shown to insane fascist religious nuts will somehow reflect badly on them IRL. This is a very short-sighted view and you can see that from how no-one who likes the Tyranids, Dark Eldar or Orks shares it. People aren't good faith going to go "OMG r U in fact a fascist because U like this?". Media literacy is bad - but it ain't that bad!
2) A much smaller and nastier/grimier group of people who genuinely use 40K fandom as a faux-ironic cover for real fascism, and unironically love the Imperium and think they are the "good guys". These people are usually pretty easy to discover in conversation because they confuse the Imperium's brand of fascism with their own, and think the Imperium would give a toss about human ethnicities, genders, sexualities and so on, which literally has never been the case, and doesn't even make sense. You see anyone claiming to be a 40K fan then saying the 40K series is "woke" because a black woman is a high-ranking member of the Imperial Guard or the like, you know they're one of these creeps. Another very easy way to spot them is that they love to get mad about the time a couple of years ago when GW put out a statement unequivocally calling the Imperium insane theofascists and stating that they are in no way the "good guys", rather than thinking it was funny and accurate.
3) People who think that 40K glamourizes fascism and, especially in this present era (transnationally) will give hope and succour to fascists and fascist-sympathizers. The problem here is, those people are not entirely wrong. This is why you are flatly wrong to insist we've got to promote stuff from 3rd through 7th - because that was a deepening nadir of 40K slipping from being cynical and satirical, into drinking its own Koolaid. There are bits of the 4th through 6th particularly that are straightforwardly glamourization of fascism and genocide, and a lot of 3rd's stuff about the Imperium verges on that. One thing a 40K show can't do is glamourize fascism, genocide, theocracy, and so on, but if it leans too hard on that middle era, it will. But as you say:
and some of the better writers for Black Library?
Exactly.
So why would you want this to be badly written? That standard of lore writing in the era I'm discussing was much lower than both 2nd edition/Epic and 8th and onwards. It's the nadir of GW's own writing about 40K (even if the BL continued on its merry way).
Which leads us to the 4th group.
4) People who simply don't want 40K subject to like, thoughtful examination by actual adults, to people thinking about how it would actually work (despite the 5% of BL authors who are already doing this). For some reason, they don't trust that it can be done justice, and want to keep it cartoonish and childish, as most of the writing in most of the codices is (you can be extremely grimdark whilst being fundamentally teenage, indeed most grimdark is sophomoric at best), so that they never have to consider an Imperial Guard going home to his family or that much of the Imperium
isn't completely horrific 24/7, but is just people living their lives under a varyingly oppressive regime until things go wrong.
This seems very silly to me, if Amazon can make The Boys considerably better than the juvenile and grimdark source material and do the same for Fallout, which again, FO3 and FO4, largely juvenile and rather grimdark (even though FO1/2/NV were inarguable less grimdark and arguably less juvenile), then why should we not hope for the same with 40K? Both shows also prove there's no contradiction at all between humour and the kind of grimdark 40K is - and there never was! Only the weird childishness of 3rd edition, where humour (and the Squats!) were excised in the name of making the game Big and Serious for BIG BOYS NOT LITTLE KIDS (whilst marketing hard to 13-year-olds, who are exactly in that mindset) thought this. It lost the
joie de guerre of the 40K setting, frankly. That's why the response to more recent editions has been so strong - not just COVID, but they're genuinely tonally far better-judged.
I don't expect a response to all of this as you said you didn't want to get into it too deeply, but the fact is, I've been following 40K since I was a child, and I didn't pull 3rd through 7th out of a hat as being the problem, so I think it's worth explaining (and again, 4th/5th/6th are the worst of it, there is an reverse-arc to it, though I'm unsure if the nadir is 5th or 6th). 3rd was the turn for the worse (despite introducing 3 races, 2 of them cool, one a massive and stupid retcon and I'm still a little mad despite collecting an army of them at one point!), 7th was where things started to improve but still weren't great.
I just want to add I will never, in my life, get over the fact that they took the Squats out of 40K because they were "too unserious" (as it were), and they took the humour out of 40K because they were getting increasingly corporate and frankly stupid since the 1994 IPO (3rd was 1998), and then put in BDSM elves and undead robots, two concepts infinitely sillier and funnier than "Space dwarves with serious firepower (and motorcycles)".
