Depends on where they come from. The game doesn't spell it out, so it's up to the DM whether they're just pulled from their dinner somewhere and enslaved, or if some god(s) of magic make deals with monsters to be on call and get paid for it. One way is evil and the other is not.
AD&D summon spells just straight up summoned any monster it could grab in the area. 3e grabbed outsiders (celestial/fiendish animals mostly) but nothing was stated that the celestial badger the wizard just summoned was a mercenary on call. I don't remember 4e summons, but 5e spells certainly seem to fit with the "grab an elemental/celestial/fey/fiend spirit and make them fight for you" system.
I guess it would be interesting to debate if the spirits that summoning/calling spells grab are actual sentient beings or just snippets of planar energy given form, but you can certainly rule that a casting Call Fey summons a pixie family to fight and die for you and you wouldn't be wrong.
his is no different than evocation. If you're using phantasmal killer to harm someone, it depends on the justifications. If you're hiding the opening of where the party is resting so you that you can be safer, it's not at all evil.
You casting an illusion to make your copper coins look like gold ones and then underpaying a merchant is scarcely better than outright armed robbery. Making yourself look like someone else to gain access to an area is identity theft. At best, it's a betrayal of trust. At worst, people have gone to jail for things first level illusionists do trivially.
And again, are you trying to steal thoughts from an innocent, or locate a kidnapped kid?
Is there much of a difference between casting Charm Person to make someone tell you their secrets or casting Detect Thoughts to rip them out of their head?
My point is that most D&D magic is at best amoral and at worst ethnically unconscionable. If charm spells are "evil" for robbing a person of autonomy, you're going to have to lump a lot of additional summoning, illusion, and divination spells in there as well. To the point where a typical arcane spellcaster could hardly be described as "Good" for knowing and using most of them.
As with much of D&D, the minute you apply real-life ethical debates to it, the game conventions collapse.