These are excellent points...I think you might have changed my mind regarding Downtime as a "pillar" of the game. There are many, many ways to play a fantasy tabletop roleplaying game, after all. Perhaps Downtime is a fourth pillar of the game: it handles a lot of interaction between the characters in the story and the setting/NPCs, has its own risks and rewards, and engages a variety of character options, backstory, and abilities.
Which makes me wonder: is there a fifth pillar of the game, Creation? Rolling up a character, writing a backstory, planning out your character advancement, interacting with your fellow players and the DM to create a role within the group and within the story...I consider character creation as much a part of "playing D&D" as I do rolling dice and calling my actions. And the same on the DM side of the screen: writing stories, drawing maps, crafting encounters, rolling up monsters and NPCs...my fellow DMs spend hours and hours each week dreaming up the adventures for their players. That, to me, is how most DMs "play D&D" most of the time.
Maybe there are five pillars of the game: Combat, Creation, Downtime, Exploration, and Social.