I think it depends.
The first question is of course whether the RPG system you're using actually has something equivalent to a Diplomacy skill.
The second and maybe more important question is: Do you, the GM want to allow the players to succeed with a diplomatic approach?
If the system features a 'Diplomacy' skill, the players want to use it and I consider it possible that they may succeed, I will ask them how they're trying to convince the NPC.
They can either paraphrase or tell me in-character. Then I'll decide on the difficulty of the check and ask them to roll.
The NPC's reaction will depend on the outcome of the check, and I'll reply to them in-character.
If the skill check was a (normal) failure, they're free to have another try, but they need to come up with a different tack.
In D&D 4e there's the covenient framework of a skill challenge to resolve more involved diplomatic encounters.
It gets trickier in systems without mechanics to resolve a diplomatic encounter - or if you decide you want to resolve it by pure roleplaying.
In that case you have to trust your gut feeling and decide when it's enough. Often you'll notice that all but one player is loing interest in the encounter.
Then it's clearly time to wrap-up and tell them they should either give up or roll initiative.
As an interesting aside, just yesterday I first read about Robin D. Laws 'DramaSystem'. I think the following is somewhat relevant to this discussion:
Dramatic exchanges sometimes arise spontaneously in the course of a traditional roleplaying game, often between player characters. What usually happens, assuming any resistance whatsoever on the part of the granter, is that the granter digs in. Both parties reiterate their positions, stalemate ensues, and the entire lengthy interaction fails to move the story in a new direction.
This happens because, unlike dramatic stories or real life, the granter has no incentive to ever give in. We have trained ourselves to think of good roleplaying as remaining true to a particular, quite fixed conception of our characters. Unexamined oral tradition tells us that it is laudable to remain static, uncompromising, even generally oppositional. Further, we have no incentive to give in. We play our characters without the emotional ties and obligations that cause us to reluctantly grant petitions in life.
(
Source)