It all comes down to the Wand. With the Wand, Orcus has the edge even without his 500 hit points of liches. If Demogorgon can get the Wand away from him, however, he's boned (heh).
Well...that depends on whether you're using the Out of the Abyss version or the nerfed Mordenkainen's Tome of Foes version. OOtA Demogorgon can exhaust Orcus's hp one round sooner than Orcus can his. But with the MToF version, Orcus still wins.
And if it comes down to trading blows? Orcus is screwed.
...
Because feeblemind is an Int save. One of the saves Orcus is not proficient in. Even with advantage, a +5 save against a DC 23 spell ain't good odds.
On trading blows, see above. The
feeblemind one though, is one of Demogorgon's best abilities. If he can use it before Orcus uses his wand,
and he can get Orcus into melee,
and you are using the OOtA stats for Demogorgon...then he probably wins. That's a lot of ifs ands and buts though.
Edit: I just figured out how Demogorgon wins. Cast telekinesis... and target Orcus's Wand. Telekinesis is an ability contest, not a saving throw, so Orcus doesn't benefit from Legendary Resistance or Magic Resistance here. It's basically a coin flip whether it works, and Demogorgon can flip that coin three times before he runs out of castings. Orcus is massively, cripplingly dependent on the Wand; without it, Demogorgon whips his sorry bloated butt all over the Abyss.
This would help, but unfortunately he can only move it 30' per round, so Orcus can just chase him down and grab it. And the real problem: it only has a range of 60'.
In general, the biggest problem is that Demogorgon's gaze is virtually useless against anyone worthy of fighting him (including an adventuring party) because no one is going to
not avert their eyes, which means he just semi-blinds some of the characters each round, if he wants to give up damage to do it. Casters are going to cast spells that don't require a direct target, and weapon users will just take the disadvantage and attack.
So let's take away Orcus' wand and figure out what Demogorgon really needs to defeat him, and just in general to be more effective.
What he needs the most is the ability to incapacitate creatures so they can't avert their eyes from his gaze attacks. He currently has no way of doing that, except through one of his gaze attacks *facepalm*.
If he has a way to do that with regularity, he could then try to smack critters out of the sky and go after them. Otherwise all they have to do is stay more than 150' from him (120' after he's blown his
feeblemind) and they are safe.
I did some math and figured out about what the CRs come out as based on the DMG guidelines:
Demogorgon (OOtA): 28 or 29
Demogorgon (MToF): 25 or 26
Orcus: 26 (wand ignored)
Zariel: 27 (for comparison with another listed CR 26 creature)
And the sad part is that you need the tough version Demogorgon just to beat Orcus in a melee
without his wand.