The potential for One Massive Encounter That Initially Appears To Be An Easy Encounter increases the tension of all encounters, even two kobolds standing around. I happen to find that cool. I don't like casual combats/killing.
Im a massive advocate for the 6-8/2 AD and I totally agree with you here. As DM you need to have the players self regulate resource use. Pushing longer AD's on them, 'one two' punch encounters like your kobold/ wave of hobgoblins, the occasional deadly level fight, and the very rare meatgrinder longer dungeon are all valid and effective tricks. Keeping your players guessing is key; if they never know whats around the next corrner - or how many more corners they have to turn before resting - then they will naturally police themselves.
Remember - the real challenge comes from the adventuring day as a whole - not from individual encounters.
That said. I am quite confident that you can design encounters in the [medium-hard] bracket to challenge any group of PCs. If the PCs are steamrolling them too easily (and remember - theyre supposed to win, expending around 10-15 percent of thier resources as they do) then re-asses why and counter that. If they rely on aerial bombardment, use more flying creatures or critters with ranged attacks (or dungeons that force the PCs to the ground). If they rely on kite tactics, use cramped spaces and have your monsters come at the party from the rear or surround them, or rely on potent ranged attacks of thier own.
Also; when designing your encounters, turn your mind to environmental challenges for the encounter. Difficult terrain, hidden pits, bridges, lava lakes, visibility issues, strange magical effects all add depth to an encounter. There is nothing more boring than repeatedly fighting stuff in a 30 x 30 stone room with nothing else going on. Better yet, if you have the time to have features of the environment the players can use against the the monsters - a huge vat of boiling stew that the PCs can push over like Conan does, pillars to hide behind, levers to push, pits to jump over or shove people in.
Lastly, change up your monsters (but CR them appropriately). Something as simple as changing weapons and armor, or adding a special ability your PCs didnt expect can break player expectations and dissuade metagaming. That Ogre that the PC's thought was a cakewalk? Turns out he recently killed an unlucky Wizard NPC and ate his potions thinking they were quite tasty; give him an ability from the potion miscability chart in the DMG and two active potion effects (he's permanently breathing fire and hasted... and very, VERY confused). Maybe make him a 'War Ogre' with half plate armor, double the HP, and multiattack. Give your Hobgoblin captain the Knights parry ability to represent his advanced skill.
Challenging and intresting environments and unique monsters make even a [medium-hard encounter] both intresting and a challenge of the players creativity and initiative - and not just an exersize in dice rolling and a challenge for the characters. Mke your players think every single encounter, and make using an ability or resource an exersize in risk/reward and a meaningful player choice.
