Charm Person is not a memory modification spell. Therefore, they'll remember being particularly friendly towards you.
That being said, the ability to identify specific spells is limited to the Spellcraft skill, and therefore most people won't know that it was specifically a Charm Person effect that made them like you.
They might suspect being ensorcelled, but they might just as well remember you as being just such a nice guy, they don't quite remember what came over them (like meeting a celebrity, where normal people go all gaga and stuff). This will probably largely depend on how they're treated during the Charm, and what kind of openings you leave them.
For instance, if you Charm Person a merchant into handing over all of his stuff at a drastically reduced rate (just because you're good friends), he'll probably be suspicious.
If, instead, you Charm Person a merchant into handing over all of his stuff at a reduced rate, secured by a line of credit with a pretty decent rate of interest, he'll likely be less suspicious. After all, that's the kind of business he does normally, right? It's just that you were so nice, he was willing to give you the "Friends and Family" discount.
