How? The game remained very compatible from the beginning all the way to the end of 2e. Lore was pretty compatible too for the most part.
The lore of 2nd ed AD&D is also pretty compatible with more recent versions of D&D.
What I'm talking about is the actual procedure and logic of play.
In the classic version of the game, the key function of exploration is for the players to be able to make informed choices about what encounters to trigger (this is the player-control over scene-framing, by choosing which doors to open and by choosing how hard to push against the wandering monster clock), and to be able to identify what loot there is to collect and remove from the dungeon (which is the game's win condition). So the ideal is a player-driven game, played by default in pawn stance. Exploration is not an end in itself - it's the key means that the players have to learn the state of the "board" (ie the GM's initially-secret map-and-key), and thus to exert the control over play that the game expects.
The game has PC-build and action resolution rules oriented towards this: rule for finding doors, listening at doors, and opening doors; rules for finding and disarming traps; potions and wands for detecting doors, traps, treasure etc (all written to work in a reasonable fashion on a dungeon scale).
Beginning in the early-to-mid 80s, and utterly consolidated by the time of 2nd ed AD&D, the play of the game has changed completely. The PC build and action resolution rules remain much the same (as you note) but for the addition of NWPs. But the way the game is actually played isn't the same at all: the GM controls the flow of information and the sequence of imaginary events. Causal connections between events are no longer mediated via the map-and-key method (which imposes a strict if artificial logic on things) and instead via the GM's imaginings about a "living, breathing" world.
why do they bother to provide an explanation for said dungeons' presence in the world at all, as the vast majority of them do?
In the classic game, it's largely a fig leaf or "lampshade" to have the fiction make a modicum of sense. But it's not more than that.
I mean, look at the intro text for a module like Tomb of Horrors, White Plume Mountain or Ghost Tower of Inverness. Look at Lost Caverns of Tsojcanth. These modules evoke some basic fantasy tropes, but they don't
actually make sense in any naturalistic way (I mean, why would Iggwilv build a treasure chamber with six symmetrical entrances that have to be opened in sequence; and if she can make doors that teleport you away when they are opened, why not just teleport you to the middle of the Sea of Dust, or to a desert island somewhere?). They are puzzles to be solved.
You can look at some late-80s/early-90s ICE modules for MERP and RM to see what a slightly more naturalistic dungeon might look like. One obvious point of comparison is that the ICE ones tend to be
more boring! Whereas whatever one makes of the WPM nonsense, it's not boring to play through.
The move from a focus on player-directed, map-and-key puzzle-solving play; to GM-directed "living, breathing" world play; is a fundamental one. It shifts the mainstream of play away from what could reasonably be called a type of wargame to something completely different. Such that, from my perspective, 5e D&D has much more in common with what you prefer, than what you prefer has in common with the classic game.
Real life with fantastic elements.
Dramatic things can still happen in a setting that mostly operates like the real world with fantastic elements. The rules or the DM certainly don't need to force that drama either, it can just emerge from player decisions through their PCs, using information the PCs have available.
On this, I agree completely with
@Hussar. The dramatic things happen
because they are written into the fiction - whether random tables that trigger (say) attacks by stirges, or the eruptions of volcanoes - or whether the GM plans for this person to do this wild thing pursuing this mad agenda, that will drag the PCs into the maelstrom.
If the only information the players have available is the sort of information that a typical person in a pre-modern society had available to them, then we wouldn't expect much drama. In fact we'd expect events to be much more intimate and mundane - but D&D is not a particularly good vehicle for that sort of RPGing. (Due to features both of its PC build rules and its action resolution rules.)