Charm spells are kinda funny. They aren't well defined in the rules for a reason. Every time you cast, the spell is different. How it will affect your target depends on the target, the caster, and how the target is treated.
Let's say I'm trying to get information about a local lord's security forces. I meet with the captain of the guard in a pub and charm him with a Stilled Silent Charm Person. (Stilled and Silent to keep the "He saw you cast" arguments away.) We get to talking, and I buy him a few drinks. He gives me the information I'm looking for because I convince him to brag about how no one's ever gotten past him, and I'm a damn nice guy whom he feels he can trust. So we part ways at the end of the evening, and eventually the spell wears off.
Now, I was very nice to the guard captain, and treated him like a friend. I bought him drinks, listened to him complain, and agreed with him that he should get better pay. I did everything a friend should do after I charmed him. How should he feel about me afterwards? The only camp I've seen so far in this thread is the "He decides he was acting oddly and gets suspicious" line of reasoning. But people will go a long way to rationalize things.
I'm of a mind that the next time I see this guy, he might be glad to see me, clap me on the shoulder, and offer to buy me a drink. Don't tell me that you've never met someone whom you were drawn to and immediately wanted to impress. Someone whom you wanted to like you. I can see charm person being like that.
Let's say that the captain introduced me to a few of his buddies over the course of the night. These guys aren't charmed. I play friendly and let their respect for the captain carry some weight. By the end of the night, I might not be friends with them, but they'll probably remember my name and nod to me when I pass them by.
The charm spell wearing off does not change the memories of the victim. They don't know why they felt the way they did, but they'll remember feeling that way. If you treated them well, they should recall you in a favorable light. You were this person who came into their life and struck them as so charming (heh) that they immediately felt that they could trust you. They wanted to impress you- wanted you to like them. If you did, they'll remember that, and probably remember you favorably.
My last example: say you had a way to charm a woman for years at a time. She was charmed and acted like she was in love with you. While, when this eventually wears off she might see your flaws much more clearly, if you two are compatable and she was treated well (and never finds out about the charm) who's to say she should immediately stop loving you?
I don't think charm spells leave a lasting effect after they expire, but I do think that the social interaction that takes place while charmed does. This shouldn't be discounted so easily.
- Kemrain the Charming.