The bolded is in fact exactly how I've always viewed it, and I don't consider it an insane theory.

So yes, the character is being taken from one world and dropped into another.
And it's not a multiverse theory, it's a
universe theory: uni means 'one'. Makes a lot of things way simpler to grok, at least for me. When my character stands outside at night on its game world and looks up into the sky, the stars he sees might very well have other DMs' game worlds orbiting around them.
I got this idea from my first DM, who was doing his masters in astronomy at the time, when he showed us a picture of a real star and said "that's where my setting's planet is". I think he even tried designing star charts and constellations based on what you'd see if looking from that location in space, though I'm not sure how far he ever got with it. I just took this concept one step further by thinking "well, if his world can be out there, wny can't all of them?", and went from there.