And your multiverse is a part of the D&D multiverse. Because you're playing D&D. That's why the goblins that appear in your world when you use the stats from the Monster Manual are the exact same goblins that appear in mine. Is that just the most unlikely of coincidences? Nope. Cause our individual worlds are connected at a meta level to D&D.
Now there might be no way for any character in your world or planar structure to use magic to GET to any other plane in some other person's world or indeed even believe some other DM's world might even exist. Sure. You can say that, and indeed you might make it a point to never have that happen in the context of your own stories. But have you ever had a player bring a PC into your game world that they've played in a different world at a different table? Or have any of your players ever taken a character they've played in your game and then played it at a game store or in Adventurer's League at some point?
If it has, how is it possible? The "same" character showing up in two different worlds? It's possible because in the multiverse there are infinite numbers of everyone and everything. And even if one of me will never ever meet another one of me, that doesn't mean that the other me doesn't exist.
And it's the same way with D&D. No one in your world might believe in the world of Eberron that Keith Baker plays or have any way to get there (because the magic or physics of your world doesn't allow for it.) But if you both happen to have Holy Avengers, and your orogs all have the same stats, and you both reference the Elemental Planes with a creature in one of them by the name of Yan-C-Bin... it's because you are a part of the D&D narrative. And no amount of denial on your part can make it not true.
How do you not be a part of the D&D multiverse? You don't play D&D.