Do you find that there's a difference in the time drag of Battlemap combat as opposed to Theater of the Mind combat?
Let's assume pre-drawn maps to eliminate the time it takes the GM to draw.
It depends on the DM. If the DM is comfortable with making snap decisions about what monsters and PCs can do - who will be caught in an area, who can reach or get away from whom, etc - and uses quick, concise descriptions; and, if players limit themselves to declaring actions that don't require a lot of checking (I attack the nearest goblin, I move to get the best shot I can and fire at the archer in the loft) and playing 20 questions to figure out what their options are (where is the goblin archer in relation to the paladin, what is he hiding behind, is it flamable, what if I move to the right, can I get a clearer shot, etc, etc, etc, etc,) then a simple combat can go a little quicker in TotM. If, OTOH, the DM describes things in detail, carefully works out the ranges/areas/movement/etc the game gives him and each player asks for all kinds of clarifications of those descriptions before deciding on their actions, it can be a lot longer.
Using a play surface gives everyone a clearer idea of the game-level scene - relative positioning, who has cover, who wold be caught in a spell, etc - if you focus on that, and the rules are fairly clear, it can actually speed things up, and a grid or hex map can further speed up the use of minis on a surface, because it replaces measuring distances and drawing areas of effect and the like. But, just as with the 20-questions players, slowing down TotM, you can have players drawn to measure or count out everything that might affect any action they're considering taking.
Bottom line, it's the players (it might take only one to slow things to a crawl) and DMs, more than the system or the mode of play, that are fast or slow.