Being able to hop between worlds with the same character every adventure would be odd and not really match most settings. It would making planar travel a little too commonplace.
Tying it to certain adventures would work, but has the potential of trapping people in unwanted settings. If they miss a game and no one wants to play the adventure where you get out of the setting, that character can’t get home.
Requiring Downtime days is an option. But if you can’t do it at will, you risk people showing up at a game for one setting and not being able to play because their character is in the wrong setting. This adds a whole other level to scheduling game days, as you need to coordinate worlds in addition to level ranges.
It’s okay for a mini campaign, that groups can play through with different characters from beginning to end. But an ongoing thing gets that much trickier.