The characters in my campaign are now rapidly increasing in power (currently 11th level) and easily capable of dishing out huge damage to opponents.
So far I've tended to go down the bag of hit points route so enemies last longer and can dish out some punishment of their own however I'm starting to think that maybe another approach is needed.
High level play creates huge design challenges. I think simply increasing the hit points of the monsters won't create challenge without undesirable sideeffects, but neither do I think what you call the 'glass cannon' approach is viable.
There are two different but important issues when dealing with high level PCs:
1) They can deliver the beatdown. This is what you are talking about. If high level PC's can unload on a single enemy, that enemy has to be extremely durable to survive even a single round.
2) They can dominate the action economy. This is what you are not talking about that you need to. In addition to beating down an opponent directly, high level PC's have the ability to steal actions from their enemies using debuffs or inflicting statuses. Combined with the fact that PC's typically have 4-6 actions to a single foes 1, this means that high level single foes will often go down hard to just a few 'unlucky' rolls.
The problem with the bag of hit points approach is that eventually in the effort to control the PC's ability to beat down a foe, you elevate the importance of the action economy too greatly. In most editions, this would result in the PC's prioritizing save or suck (or save or die) attacks, and consequently elevating the importance of spellcasters.
The problem with trying to address this with a glass cannon approach is that you might still end up elevating the importance of the action economy while risking the problem of one shotting PC's, which is generally no fun tactically or thematically. Keeping PC's on the verge of death provides tension and allows them to make choices. Winning the initiative and then randomly killing a PC before they can act doesn't make for fun for anyone.
In general, the best way to challenge high level characters is with more foes. The biggest mistake I see most DM's doing is trying to challenge high level PC's with a single foe all the time, when in fact this should be a relatively rare situation and requires very delicate handling. One of the best things 4e introduced was recognition of this fact, and though I didn't always agree with the specific implementation, the fact that 4e was trying to introduce formal design approaches to make encounters and particularly encounters with single powerful opponents interesting (especially at high level) was a good thing. 5e to some extent follows on this approach. If you are using 3e, recognize that most high level foes are relatively badly designed and require very specific handling and/or redesigns to make them more interesting.
Having multiple foes deal with most of the issues of high level design handily. The PC's are no longer as easily able to dominate the action economy. The monsters are generally no longer as able to one shot a PC, and not only can the DM reasonably narrate why the NPC's are focusing fire on a single target (lack of coordination, each monster attacks its most accessible foe, etc.) but often as not simple geometry prevents that sort of focus and the PC's can use formations and terrain to prevent that focus. And the monsters generally have a big bag of hit points that makes them hard to kill quickly, often far more hit points than any one single foe would have. The are still some design challenges in selecting the foe(s) you wish to use to challenge a high level party: they should have a ranged attack, they should have some defenses against common area of effect attacks, they should have an attack that is reasonably level invariant (touch attacks, save for half, very high attack bonus, automatic damage, etc.), they should have enough HD to avoid HD based win buttons, and so forth but in general this is an easier approach in most additions.
For a single foe, you really need something that has multiple actions each round, and preferably at least one being an area of effect attack. The goal here is be able to inflict lots of damage, but also be able to spread it out. A monster with both an attack and a damaging aura or gaze attack is a good example, or a monster with both an attack and the ability to use a breath weapon or a spell like ability as a free action. Alternatively, the monster can have many attacks, like a hydra, but have the monster specifically limited to only a certain number of attacks on each foe.
Monsters with 'reactive' attacks that can take place on the PC's turn also qualify here - often these are attacks triggered by attacking the monster. Examples are, successfully hitting the monster sprays acid blood, or raises a cloud of choking spores, or causes the weapon to become stuck, or perhaps the monster is covered with spikes or flames so that an attack draws a counter attack. Alternatively, the monsters body could be super hot or cold, resulting in heat/chill metal effects when you touch them. In some editions like 3e, having Combat Reflexes and melee attack with reach is also a good example of being able to reactively counter the PC's. Another idea would be having reflective armor that bounced spells if the monster's Spell Resistance (or saving throw) succeeded.
The single monster also needs to be able to threaten to steal actions from the PC's either by forcing them to move, forcing them to aid other PC's, or taking the PC out of combat for a round or two. This can involve a monster that skirmishes successfully and forces PC's to spend rounds chasing, or it can be a monster that has short term area based debuffs, or it can be a monster that can create terrain or obstacles over the course of the fight.
And this is on top of the monster needed to be durable enough to survive to use these abilities. At some point, this means more than just a bag of hit points but also the ability to resist or shake off debuffs, mitigate against spells, and possibly even mitigate against bad luck with some sort of 'destiny point' mechanic or some sort of 'on failure' mechanic.
A single foe doesn't necessarily need all of these abilities, but the higher the level of the PC's the more of these abilities a single foe has to have in combination to be an interesting challenge.
If you are playing 4e/5e, the designers tend to have gone at least halfway to where the monster needs to be with concepts like 4e's 'solo' foe or 5e's 'legendary' foe. If you are playing 3e or earlier, you are probably going to have to do a bit more thought when thinking about how to present a single tough foe.