I don't think it's about the fantasy power level per se, I think it's about gamism vs. simulationism. You're basically gamist, as I understand it, and you try to provide fun encounters to your players. (Which BTW is perfectly fine. Not my cup of tea but not badwrongfun.) I am simulationist and try to provide a fun, coherent, interactive world for the players to generate their own conflicts/encounters within. At low levels, memorable and difficult fights in my campaign have consisted of things like getting detected by flying apes and draconian cavalry (re-skinned Centaurs) while sneaking into enemy territory, and holing up behind boulders for cover while trying to repel their missile fire. (Again, I did not force this encounters on them since it's a sandbox. I telegraphed the threat in this area, and the NPC's request/demand, and the players chose to try to infiltrate here instead of going elsewhere or gathering an army or negotiating or anything else.) A squad of a couple dozen cavalrymen isn't exactly a high-powered fantasy trope. The only thing it's violating is the DMG encounter guidelines, which is a gamist thing and not a fantasy convention.
DMG guidelines at low levels exist to generate squads of hobgoblins which are just barely big enough to lose in an interesting way. Simulationism dictates that squads of hobgoblins should be small enough to maintain a good force-to-space ratio over contested territory but big enough to not needlessly endanger the force against expected opposition. How big that is depends on a number of factors--and so there are multiple paths to victory. You can either hammer straight through the hobgoblin force, or you can start ambushing little chunks of two dozen of them at a time until they contract into a big ball of 400 nasty hobgoblins all in one fort--and then you bypass and ignore them. Mission accomplished, depending on what your mission was. To me that makes 400 hobgoblins an interesting challenge at mid levels (say, 5 to 8). At low levels (1 to 4) it would be a deadly challenge but you might have to engage anyway, cautiously, if all the other good guys are dead. That is an interesting challenge in a different way.
I don't acknowledge gamism versus simulationism. D&D is not like most games and isn't close to being a good simulation of the real world.
I look at D&D as cooperative story-telling. The DM creates a loose plot within the setting he chooses informing of the players of the parameters of play to guide them in the character creation process. The player's create characters they believe would be fun to play within the scenario the DM has outlined. They are the characters in the story, the DM is the plotter and world controller setting plot points in front of the players within the given world allowing them to take action to resolve each plot thread laid before them. He does this with a combination of challenges tailored to the characters the players have created and the general preferences of the players. You create this cooperative illusion of adventuring in a fantasy world as heroes (or sometimes villains) whose decisions affect some part of the world (or the entire thing) in some substantive way. I take the greatest pleasure when a story unfolds close to the manner I planned with wild card factor supplied by the decision of the characters that may take things in an unexpected direction. I tend to focus on the story using the mechanics to create the scenes and results I want, while always maintaining the illusion that the characters are "real" for the players.