This seems no different than considerations for wizards or sorcerers or bards. They have magical power and are not constrained by a patron or predictable based on their patron.
Well yes, spellcasters are widely distrusted by the non-magical world, because their powers are strange and dangerous. A wizard could make you believe that you've accepted coin from him when he'd only handed you a bit of tin or brass, or he could turn invisible and go about your home, or eves drop on your private affairs with his magic glass, or charm your daughter to make her believe she is in love with him, or consort with evil spirits do avenge himself on his enemies, or raise the corpse of your mother up to do his biding. In short, even in places where magic is considered a trade, it's often considered a dirty and dishonorable trade greatly to be distrusted.
In my campaign, sorcerers in many parts of the world are considered not even human, and are therefore subject to the same "free to slaughter" considerations that would generally apply to vampires, werewolves, or marauding dragons. The fact that they can perform magic without resorting to lore and study, proves that they aren't truly human, but monstrous beings with tainted blood, twisted and accursed souls, and perhaps even evil spirits in human form.
Bards have "secret colleges" because they literally have to keep their practice secret. Many people aren't even aware bards, and especially human bards exist. And, if they did, they'd probably murder them. Bards that don't keep their magic secret, have to find some socially acceptable excuse for it - which could depend on the community. ("I'm actually a fairy!", could work in some places, as it's accepted that fairies have strange magic Of course, this presumes fairies are acceptable, which in some places, they aren't.)
Wizards are acceptable in about 3/4's of the world, though there are places they are treated as evil. The reason for both are similar. Wizardly magic nearly destroyed the world in the past, but it's believed in the more tolerate regions that after that happened and the culprits slain, that the current practice of wizardly magic is the lore that the gods left alive in the world for the benefit of the free peoples, and all the really nasty stuff ("art magic") was censured and erased leaving only "spell casting" and a bit of alchemy behind. They are reasonably heavily regulated by society, social conventions, by certain cults, and by their own members. They in return also get certain social privileges - such as the right of privacy (a declared wizard may not be randomly searched in either his person or his home), and are entitled to honorifics ("His Potency, the Wizard Galforth"), and are immune to certain taxes and tolls.
Anything that increases distrust amongst the general public and especially the ruling authorities is a threat to the whole wizarding community, so it's everyone's problem. They also tend to follow an unwritten rule to not get too heavily involved in politics, because there is general belief that while the gods wouldn't intervene to stop a wizard's private affairs, if a wizard gets too public in his affairs the gods might squash him. Most wizards are also very careful about sharing their secrets for a similar reason - they are afraid the gods will see them as trying to recreate the Age of Wonders when the art mages made powerful magic ubiquitous. While there are places where the streets are lite by continual flames or have a ruling class of wizards, much of the world considers that just an invitation for disaster.
I typically start campaigns in reasonably tolerant areas of the world just because I figure players will want to play spellcasters, and players will be unused to how spellcasters are often treated. Still, there has been incidents where the player's got in trouble with the law. There was a death warrant out for a PC sorcerer for a while, which the PC only evaded when the rest of the party proved that the woman who brought the charge of witchcraft on the PC was herself actually a witch. If the authorities had known that the PC was what they were, they would have killed both of them. That PC died, but the current PC sorcerer has an Inquisition from one of the major temples chasing the PC down with a Bull of Anathema, that decrees the PC is not human. The party actually killed one of the cults "paladins" that was in the process of trying to exact said degree with extreme dispatch, and the PC cleric in the same group has been advised by their temple that they think the Bull may have merit, and the PC cleric has agreed to keep an eye on the PC and watch for signs of evil behavior. In character, there are at least two members of the PC party that if they thought the PC sorcerer was in some fashion evil, would probably try to kill him - and they are beginning to have their doubts (for perhaps very good reasons) with the party Shaman, which just sold part of her soul to Urglick the Stinking Beast in order to save the life of the "Paladin" - that would probably kill the Shaman for doing so if he knew that it was done and understood what it meant.