Micah, I understand from my own experiences that world-building can indeed be a fun exercise.
However, I don't feel like I am playing D&D (or some other TTRPG) when I do worldbuilding. IME, it was quite the opposite experience. There were many times I found myself world-building for some TTRPG precisely because
I wasn't playing the game. It was all I could do in the absence of a gaming group. I may have still been engaging in the general hobby of TTRPGs by doing so, but the idea that it constituted "playing the game" is pretty absurd to my experiences. If you had told me that I was playing the game, I would have been pretty insulted, frustrated, and hurt by that idea, because in no way did it feel like I was in any way, shape, or form playing the game despite wanting to play. In retrospect, my world-building was mostly an exercise of self-indulgence.
These experiences world-building in the absence of a group plus similar experiences to what
@Mort recounts of "setting tourism" helped me realize that world-building is pretty pointless self-indulgent exercise in the absence of actual play. It may be prep, but I wouldn't call it play.