Saeviomagy has it right. Without Spellcraft, the ex-charmee might never figure it out. Even with Spellcraft, the ex-charmee might justify it to himself, like, "Well, sure he cast a Charm spell on me, but I liked him anyway."
There once was a girl I was quite enamored with. Now she's long gone...