Is there any system anywhere that's figured out a good way to keep track of player's alignment?
I do it informally. After watch a player play for a while, if they can't hold to the alignment on their sheet, I'll give them a knudge reminding them that there character is X alignment. If they still can't hold to it, I'll suggest that they consider changing their alignment to what I think they've been playing the character as. That's usually not a problem except with players playing a divine class of some sort with alignment behavior restrictions, in which case I start handling it via in game reprimands/chastisement from their patron and we have IC discussions of what it means to serve a particular deity.
Our party is TN or XG at the moment, but one of them (our ranger) likes to do things that are... evil... like scalping enemies and wearing their ears as trophies in order to improve intimidate.
You are going to need to start defining what it means to be good in your world. In my world...
a) Fundamentally I consider the act of scalping a dead enemy to be neutral. It holds no particular moral value in and of itself, though it could be a mark of some very depraved values. More on that later.
b) Scalping enemies would break the social strictures (law) of most civilized nations (at least, the ones that think of themselves as civilized). This would be considered a 'barbaric' act and would not be socially acceptable. Amongst some other cultures, taking scalps would be approvable. You could be a member of that culture and still be lawful. The lawful member of a 'civilized' culture might not approve of your particular action, but might be able to sympathize with the lawful justification you had for your action (for example, obtaining honor and chastening the foes of your community).
c) Scalping a living foe is an act of unnecessary cruelty, and so evil in my game world. Scalping a dead foe to obtain a trophy is vain-glorious, and so chaotic at a minimum and in all likelihood evil. Scalping a foe because you enjoy and relish the violence is evil. Scalping a foe to ritually humilitate a dead foe is evil. Scalping a foe to obtain power over their soul by making a fetish of the body part is evil. Scalping a foe in order to achieve honor and social acclaim is lawful, but likely lawful evil because of the societies clear glorification of violence. Scalping a foe in order to intimidate or frighten foes in order to protect yourself or larger society is dispassionate and therefore neutral (with law or chaos according to the group you are protecting), but scalping a foe in order to more greatly enjoy terrifying your foes and to make yourself fearsome in your own eyes and the eyes of your foe is chaotic evil (again, glorification of violence, this time with respect to self).
d) In all all cases, scalping a foe is morally dangerous because of the slippery slope you put yourself on by engaging in acts of prolonged and meditated violence. The ratio of necessity to violence here is really low. It's going to be hard for a person to avoid slipping into depravity and becoming totally densensitized to violence. This is a danger for any good person engaging in violence, and why most good aligned people would shun acts like scalping. If the act was socially acceptable, a good aligned member of society might not initially recognize the danger and might be more resistant to thinking about it, but the more the act sticks out as aberrant, the steeper the slope the character is on because it is unavoidably a thoughtful and conscious act.
If the character is already neutral, then I'd say this is a good indication he's slipping toward evil. I'd also, from a meta-perspective, consider an act like that a clear indication that the player was wanting to explore playing an evil character. I'd first privately talk to the player about his intentions. If the player wasn't really conscious of what was going on, or disagreed about your moral analysis, I'd make sure he understood that your world's morality wasn't necessarily a complete reflection of the real world. I'd make sure he understood that there needed to be a concrete arbitration of what was good/evil, and that it's understandable if his understanding wasn't completely congruent to the game world but the game world's internal definitions trumped anything else. I'd make sure the player understood I wasn't passing moral judgment on the player, but inform the player that he was on a slippery slope and I'd be looking for other markers of evil in his character's actions and that, if these became common or severe enough I'd have to alter his character's alignment accordingly.
It's always better though to work with the player on these things.
It's also worth noting that the action, regardless of its moral value to the character doing it, may be morally abhorrent to other members of the party.