A suffusion of yellow
in WandaVision we get the scene where Wanda comes out of the Hex and tells SWORD to leave her “and her family” alone and later we get the scene where she Purposefully expands the Hex turning the SWORD camp into a circus - that told me that Wanda is aware that She controls the Hex and the town-illusion, but She deludes herself into living in her fantasy. She may not have been malicious but she did make a choice to cause harm to its residents.The difference is that there's no sign the Purple Man had any ability to recognize he'd done anything wrong, which was clearly not the case with Wanda. Wanda, to the degree she harmed people, did it with no real intent; the Purple Man did it not only without concern, but often maliciously.
If people are gonna make analogies here, you need to at least compare like to like.