First the party has to figure out where the ogre mage is in order to glitterdust.
Fortunately, Spot and Listen allow you to notice and pinpoint invisible creatures, respectively.
Then the ogre mage has to fail its save, which depending on the character we can count on being between 15 and 16, not a sure thing at all.
It's a 70-75% chance of the Orge Mage sucking due to the first spell cast. I call that pretty good.
Further, even while blinded, the ogre mage is no slouch. He can still fly out of reach.
And is no threat to anyone if he's retreating.
In fact, he can cast darkness, equalizing the situation.
I cast Magic Missile at your Darkness.
Now, against a party of well-armed fighter-mages armed to the teeth with glitterdust and magic missiles, the ogre mage is going to be in trouble, but against a more conventional party, he is significantly threatening. Against a high level party, the low Will save is more of an issue.
Round 1: Rogue notices something and marks the area for the wizard to glitterdust (say, by running up to it, or shooting in the square) Wizard does so. Ogre Mage is revealed and has a 70-75% chance of being blinded.
Round 2: Orge Mage either tries to escape or Cones of Cold the party. In the first case, the Orge's efficacy is diminished. In the second, he's marked and will be subject to attacks. Wizard can cast Glitterdust again for another 70-75% chance to blind it if necessary. (Admittedly, a good Cone of Cold could take out the party wizard because wizards don't have a great deal of HP).
Round 3: ???
Round 4: Profit.
Now, unless parties with Rogues who have invested in Spot and Listen, along with Wizards who use Glitterdust, are now "high powered parties", I think we can consider your point disproven.
CR 6 was put forward as a good benchmark. Well, I'm picturing a 6th level party, and I frankly doubt they are going to fully prepared for an invisible, flying foe that can melee and has a very nasty cone attack.
Unless they have anyone with a decent listen check (Ranger, Rogue, Barbarian, Druid).
At that level, they could literally run out of glitterdusts before they ever found him.
How fortunate it is that we can find out which squares invisible creatures are in without resorting to random blasting.
If the fighter closes, the ogre mage is not likely to offer a full attack. And with a +12 grapple modifier, what's to stop him from pinning a 6th level fighter, flying straight up in the air, and dropping him?
1. It provokes an Attack of Opportunity from the Fighter because it does not have the feat Improved Grapple.
2. The fighter will have a +10 grapple modifier from strength and BAB, which means the orge mage has a 60% chance of success. This is lower than the 70-75% chance of it resisting Glitterdust.
3. It has to initiate grapple the first round, and then pin in the second. This means it can be sneak attacked in the meantime by the Rogue.
4. Opposed grapple checks to move. Again, 60% chance of success.
5. Wizard casting Grease or Enlarge Person on the Fighter. Grease helps him escape, Enlarge Person gives him a +4 bonus to grapple... which makes his total bonus +14.
6. The cleric can force a will save vs
Bestow Curse,
Hold Person, etc. This would be bad for the Orge Mage.
7. You move at half your speed when grappling after winning the opposed roll. Ascending also forces you to move at half speed. This means in one round, the Orge Mage can move up and drop the fighter... 10 feet. This deals 1d6 damage. Using two rounds to pursue an attack that is unlikely to work, provoking several attacks of opportunities, all in order to deal 1d6 damage does not strike me as impressive.