I have two moments that really stand out, but I'll start with one.
D&D 1st ed, heavily houseruled. Solo Play (the Wife ran a paladin and a Wizard). The characters are high teen levels IIRC. And the wife has played Thonolan her Paladin perfectly since level 3 (which was where we started). The character was courteous, kind, stern when he needed to be, and relentless in his pursuit of evil, and also relentless in his pursuit of helping those less well off than himself. The perfect paladin.
"I want to run a fallen paladin story for Thonolan" the wife says.
I just stare at her. "You what?"
"I want to run a fallen paladin story" she repeats.
So I think about it. The character is a Paladin serving Athena, the goddess of Wisdom, Tactics, and defensive Combat (as we ran the Goddes in our games).
So I set up one of those "chose chaotic or choose evil" type things that bad GMs throw at paladins specifically to make them lose their paladinhood. Thonolan chose the perfect solution, the action which was least in conflict with his code, but created a greater good. He loses his paladin powers until he can get to a priest and atone. It is refused.
So the character is feeling rather put upon, and gets all melancholy, turns into a near drunkard, and hooks up with some mercanaries. The alignment of the mercs is all over the place, but the leader is evil. Thonolan doesn't much like him, and some politics happen with parts of the troop lining up behind Thonolan and the rest behind the leader. A showdown will happen. All of this over the course of a couple of levels - a few months in real time.
Then the leader tells them of a great contract, that will pay lots, and ally them with a power going somewhere. The group goes to the meeting, and it is a temple of Ares (Ares and Athena were in a cold war, with some open battles happening). Thonolan balked, and refused to work for them. The showdown between them happens, and Thonolan is the victor, and gets healed by the group cleric.
Being the new leader he tells the High Priest of Ares that they are no longer interested. The priest is annoyed and wants his advance back. They give it to him. Then the priest attacks, along with other members of the temple. He is killed rather easily, but his dying breath summons an avatar of his god. (We used a lower level writeup based on the 1st ed DDG).
He offers Thonolan service (effictly doing the blackguard thing two editions before), willing to take him in as an anti-paladin. He was sorely tempted but refused. The two fought. Thonolan won but with single digit HP (and really great tactics from the wife, and really great use of some of the special abilities he'd acquired (the house rules mentioned above)).
The avatar of Ares is defeated. We had established in the world that this means that he cannot appear as an avatar on this world for 101 years.
Then Athena shows up. She tells Thonolan that this was all a ploy of hers (Goddess of Wisdom and Tactics). She saw that something like this could happen, and forced him into a mission that would, by the formal rules, make him lose his paladinhood, and then denied his atonement, to set him adrift. She then almost tearfully tells him of how much she hated to do this, but doing so let him get close enough to Ares to kill the avatar, something he could never do as a Paladin, and that she trusted him to remain loyal, even when it looked as if she abandoned him.
And he did stay loyal.
She restored his paladinhood, and then more.
The wife was really pleased with the story, and I was amazed as a GM that I could pull that off, especially the ending that kept the character in the good graces of the Gods.
The more was multiclassing to Cleric, and some special abilities borrowed from the old "Saints" article from Dragon Magazine.