I dunno if a prestige class is the way to go. Nightmares have a cohort level adjustment of +4 and 6 hit dice, making them suitable as a cohort for a 14th level character- arguably where the prestige class should first become accessible.... and fairly high for a prestige class
Moreover, the nightmares abilities- particularly etherealness and astral projection are pretty dang powerful (If you do decide to allow this as a prestige class, at some level- a rider should gain the ability to share these abilities with its mount).
If I were to do it- I'd make the prequrisite something like the character must have a nightmare who will willingly serve/obey/cooperate with him. If the character has this, either through binding the nightmare and making a deal, or earning its trust, or whatever- then some abilities should become accessible.
I'd make it a 3-5 level prestige class... with abilities including the share Astral Jaunt/Etherealness... immunity to the nightmare's smoke attack (and the ability to see through it so there's no concealment)- and MAYBE a bonus hit dice or two at the highest PrC level. Nightmares have a decent armor class, and tons of abilities- and with their flight and spell-like abilities- won't have a problem fleeing if things go bad. The only real negative is the lack of HP... so maybe an ability that the nightmare can use it's own HP, or half of its masters (whichever is higher).
Remember, nightmares are sentient- smart, wise, and very very evil. I'd be wary of treating them like just another mount- they need to be seen as another NPC- not neccesarily completely under the player's control.
Prerequisites... maybe leadership feat, ability to have a nightmare as a cohort, making deal with said nightmare, any evil, mounted combat (and possibly some other mounted feats), knowledge the plane 7 ranks- Ride 10 ranks... with the ability to lose all these powers if you anger said nightmare, and the powers staying gone till you re-unite with him or find a new one.
I suggest a good BAB bonus progression, and maybe good will saves.
That's just stuff off the top of my head- hope it helps.
Vorp