B. The AD&D Rule Books Expand on the Multiverse
The Gygaxian multiverse was then slightly expanded upon and codified in the AD&D Player's Handbook (1978). Of importance we see the following in the PHB:
There exist an infinite number of parallel universes and planes of existence in the fantastic "multiverse" of ADVANCED DUNGEONS & DRAGONS. ... The Prime Material Plane (or Physical Plane) houses the universe and all of its parallels. It is the plane of Terra, and your campaign, in all likelihood. ... The Ethereal Plane is that which surrounds and touches all of the other Inner Planes, the endless parallel worlds of the universe, without being a part of any of them. (PHB 120).
The Dungeon Master's Guide (1979) was more explicit. The section "TRAVEL IN THE KNOWN PLANES OF EXISTENCE" states the following:
The Known Planes of Existence, as depicted in APPENDIX IV of the PLAYERS HANDBOOK, offer nearly endless possibilities for AD&D play, although some of these new realms will no longer be fantasy as found in swords & sorcery or myth but verge on that of science fiction, horror, or just about anything else desired. How so? The known planes are a part of the "multiverse". In the Prime Material Plane are countless suns, planets, galaxies, universes. So too there are endless parallel worlds. (DMG 57).
The DMG explicitly states that alternate planes in the Prime Material do exist, and states that other game systems that aren't fantasy (such as Boot Hill & Gamma World) can be used. There is even the concept that some planes would allow for breathable atmospheres between the "main planet" and the other planets and moons - someone was thinking of Spelljammer!
Later on, the DMG makes it explicit, with conversion tables to allow characters to go back and forth between D&D, Boot Hill, and Gamma World, and explicitly states that all possibilities of adventure are contained within the worlds of the Prime Material.
Similarly, there are places where adventurers can journey to a land of pure Greek mythology, into the future where the island of King Kong awaits their pleasure, or through the multiverse to different planets, including Jack Vance’s “Planet of Adventure”, where they hunt sequins in the Carabas while Dirdir and Dirdirmen hunt them. (DMG 112).
In essence, what was being codified in the text is what was already happening- tables were using D&D to play in all sorts of ways, and going between all sorts of fantastical worlds.