I'm kinda groping for an idea of what an appropriate level of power would be. I keep coming back to the notion that what is very difficult at level X should be fairly straightforward at level 2X, and easy at level 3X or 4X. Or what is straightforward to accomplish with a creature of CR Y should be very difficult to accomplish on a creature of CR 2Y and impossible with a creature of CR 4Y.
So an 17th level caster might find it quite an undertaking to call a CR 20 balor and securely bind it; the same degree of difficulty and uncertainty should beset a 25th level caster who wishes to similarly constrain a CR 30 demon prince. But at level 34 the mage might find a balor to be a useful servant and compactee (though one that needs to be watched carefully); a level 50 wizard may find the demon prince a similarly useful, albeit dangerous, underling.
This raises the question as to what feats and equipment one should posit for very high level wizards. For wealth I rely on Upper_Krust's cubic wealth formula; gp is level cubed times 100. For feats I am thinking mostly Epic Spellcasting and Automatic Metamagic Capacity, with a few other feats. A ratio of 1

1, perhaps. Maybe 1

2, maybe something else. The other feats are anything from leadership to item creation to specialized feats that may be (but are probably not) relevant to the task at hand. Self-Incarnation and Dream Walking for a Fillein type, perhaps; neat, but not relevant to bindng demon princes. A character's interests and focus will move on, anyway; after the current BBEG is soundly defeated, an entirely different challenge may await, and one's old specialty may have to be adapted to new exigencies.
I would prefer to start at the top and work down; to figure out if it would be appropriately powered for a character of level N to cast this spell, and then trust that the cost of the mitigating factors will make it appropriately difficult for characters of level N/2.
A level 50 character will have 25 more skill ranks and 16 more feats than a level 25 character. Say he has King Solomon's Seal, 7 AMC feats, 3 more Epic Spellcasting feats, and other feats irrelevant to the undertaking (specialized feats in other areas, perhaps). That's a USP of 77 (53 + 10 KSC + 14 AMC) that he can achieve without mitigation (I'm ignoring the save and SR bonuses for the moment, since they aren't at play in our seed.) His target is around 110, though, since his spell is not restricted to one fiendish noble, and his compel is not so limited (more a
charm type effect to speed diplomacy and ensure honest dealing; negotiation and payment is still involved, and the noble probably wouldn't break the deal if the compel were prematurely ended; the [compel] is a convenience, not a necessity). With an hour casting duration, some power components (100 000 gp would not be a major expense at this level) and the aid of a few assistants it should be straightforward.
But there's nothing wrong with this kind of thing going on at 50th level; and so I'd hazard that the spell, heavily mitigated to put it into the hands of a 25th-30th level wizard and his cabal, would also be OK.
[edit]
I'm not a big fan of the whole
dominate monster thing. If the number of thralls is limited (you have to fill a cohort slot or something) then that might be one thing, but it seems likely to destabilize a campaign (if your slaves are of a higher CR than their master) and render role-play superfluous. Though a spell that facilitates bargaining with stubborn or hostile fiends would be nice; you'd have to back it up with a fair deal, though, or risk the fiend reneging on their side of it.
I like how Ortwin's servitude played out- maybe make the basis a
charm monster effect, or maybe an initial period of
domination followed by days of being
charmed? Or maybe tie the effect of the spell to its SP; you can't long-term
dominate a creature unless the SP of the spell is at least 2 higher than its CR.
Speaking of goetia, I imagine you saw this class I posted some time ago. The flavor text should be familiar:
[sblock= Goetic Magician]
From "The Ethical Use of Arcane Magic: an Oronthonian's Guide," by Deacon Rhodin of Iua.
"Beware the temptations of Goetia, for those who would use diabolism to achieve their foul ends, our Lord has no mercy. Pain and suffering immeasurable shall be their lot, as their souls are condemned to the pit. There they will be immersed in great lakes of boiling lead, until the last days."
Roland’s discourse continues in a similar flowery and rhetorical vein for several pages, admonishing the true believer against using dark magics and citing numerous theologians to back up his point. Further into the chapter, beneath a stylized plate of a wizard fleeing from a horned demon, Roland finally addresses the nature of Goetic magic.
"What is Goetia, you may ask? It is the greatest peril. It is dealing with fiends to achieve your ends, and claiming that your ends are good. Only the purest and most stalwart of souls may endure such vileness without the taint falling upon them. Are you one of these? I doubt it.”
Several magical diagrams follow, accompanied by descriptions of summoning rituals.
A goetic magician is an arcane specialist who follows a strict code of conduct when dealing with fiends.
Hit Dice: d4.
Requirements
To become a goetic magician, a character must fulfill all the following criteria.
Knowledge (the planes): 6 ranks
Knowledge (religion): 6 ranks
Diplomacy: 6 ranks
Feats: Augment Summoning, Skill Focus: Knowledge (Religion) or Skill Focus: Knowledge (the Planes), Spell Focus (Conjuration)
Spell casting: must be able to cast
dimensional anchor, lesser planar binding, summon monster IV (or higher) and four arcane enchantment (compulsion) spells, at least one of which must be of level 4 or higher
Languages: must be able to speak Abyssal and Infernal
Class Skills
The goetic magician’s class skills (and the key ability for each skill) are Concentration (Con), Craft (Int), Diplomacy (Cha), Intimidate (Cha), Knowledge (arcana) (Int), Knowledge (the planes) (Int), Knowledge (religion) (Int), Profession (Wis), Sense Motive (Wis), Spellcraft (Int). See Chapter Four: Skills in the Player’s Handbook for skill descriptions.
Skill Points at Each Level: 2 + Int modifier.
Level...BAB...Fort.....Ref….Will... Special.......................Spellcasting
1.........+0.....+0......+0....+2.... fiendish luck, goetia....+1 level of existing spellcasting class
2.........+1.....+0......+0....+3.... unholy bargain............+1 level of existing spellcasting class
3.........+1.....+1......+1....+3.... goetic initiate.............+1 spell power
4.........+2.....+1......+1....+4.... hard bargainer............+1 level of existing spellcasting class
5.........+2.....+1......+1....+4.... the will and the way....+1 level of existing spellcasting class
6.........+3.....+2......+2....+5.... goetic master.............+1 spell power
Class Features
All of the following are class features of the goetic magician prestige class.
Weapon and Armor Proficiency: The goetic magician gains no additional weapon or armor proficiencies.
Fiendish Luck (Su): Once per day the goetic magician may reroll one ability or skill check; the better result automatically applies.
Goetia (Ex): The special abilities of the goetic magician class are known collectively as goetia. If a goetic magician uses goetia when summoning or calling a creature, and then instructs that creature to hurt, kill or oppress a non-combatant sentient being of good alignment, future use of goetia will be corrupted. Usually this means that conjured creatures will no longer be completely under the goetic magician’s control.
The degree to which conjured creatures will disobey or try to harm the goetic magician will be similar to the way that the goetic magician abused her powers, but to a greater extent. If the harm is slight and indirect (due to careless instructions, say), conjured monsters will thereafter be unruly, and disobey in minor ways; if the harm is direct and explicitly ordered, they will become completely uncontrolled. A similar corruption of class abilities will occur if the goetic magician fails to fulfill a bargain with a fiend. These penalties can be removed only by fully repairing the harm done (where possible) and making additional reparations (generally three times the value involved in the breach of the code of conduct).
Unholy Bargain (Su): The goetic magician may, in place of any planar binding spell, cast the corresponding planar ally spell instead; the target creature may still be specified by the caster, but is more cooperative. Only evil creatures may be called, and they react to the goetic magician as if she were of similar outlook and ethos.
Goetic Initiate (Su): When a goetic magician of 3rd level or higher casts a summon monster spell, it is treated as the equivalent spell of one level higher for purposes of summoning evil creatures. For example, a goetic magician refers to the summon monster V list for evil outsiders when casting summon monster IV. Furthermore, when casting any planar binding or planar ally spell, she can call an evil creature with 3 HD more than is normally allowed.
+1 Spell Power: This ability increases the goetic magician’s effective caster level by +1 for purposes of determining level-dependent spell variables such as damage dice or range, and caster level checks only. She does not gain additional spell slots or spells known.
Hard Bargainer: At 4th level the goetic magician can bargain with extraplanar creatures so that the payment for services rendered is only 50% the standard fee.
The Will and the Way (Su): At 5th level the xp costs for any conjuration spell the goetic magician casts are reduced by 50%. When she is otherwise capable of casting 8th level spells, she automatically adds Lesser Gate to her list of spells known. Lesser Gate is an 8th level spell that can only use the “creature calling” function, but which is otherwise identical to Gate.
Goetic Master (Su): As Goetic Initiate, except the goetic magician refers to the summoning list two levels higher than the one for the summon monster spell she is casting. Furthermore, when casting any planar binding or planar ally spell, she can call an evil creature with 6 HD more than is normally allowed.
------
Comments:
Fiendish luck is in case you roll a 1 on your charisma check to keep a fiend bound. Planar binding hit dice caps end up being the same as in the old 3.0 rules.
Lesser gate is typically achieved about the same time that other spellcasters are able to acquire
gate, since their spell advancement is two levels behind.[/sblock]
The class benefits could probably be extended to epic levels, but I wonder if it might be broken. Goetic Master gives a -6 mitigating factor for calling fiends; the Will and the Way allows xp to be a preferred mitigating factor; hard bargainer makes it cheaper to strike bargains. The loss of two levels of non-epic spellcasting, the feat requirements, and the behavior code of goetia- I don't know if it is sufficiently balanced.
I'd appreciate your feedback.