fearsomepirate
Hero
Who are you hostile against?
As an agent of hell who has been captured and bound to this craphole of a world, Nightmare defaults to hostile against all beings on the material plane he doesn't have any good reason to be friendly with.
Who cast a magic against him?
Planar Binding has VSM components, including a jewel worth 1000 gp. Subtle Spell only eliminates V and S, not M, so your use of the jewel was entirely perceptible. Regardless, he knows he was bound this plane and subjected to your orders. He dislikes that (and you). He does not need to deduce when and how the spell was cast to be angry.
Furthermore, this is irrelevant, as he is now hostile to both of you. Should a random beggar appear on the ground beneath you, he would be hostile to him, to. He's a demon. They're not nice.
That's my ruling.
It is now your turn. Tell me what you do, and I'll give you the dice rolls.