I don't worry about Skynet or Roko's Basilisk-style AI. Those types of AI will probably only exist if we purposefully make them or screw up a ridiculous amount. I'm more worried about the effects that the loss of jobs will mean for humanity. Eventually, there will stop being "new jobs" for humanity to do that AI and automation cannot do. AI will even be better than us at activities that tend to be thought of as "human-exclusive", like art, creative writing, music, voice acting, and film-making/animation.
What will we do when every part of a movie, TV show, or book can be automatically designed by an AI better than a human can? Novelists, artists, singers, and actors might not even be careers in a hundred years because of AI. If an AI can write 20 high-quality, engaging books in a single day, authors will go out of business. If an AI can animate a whole TV show and "voice-act" all of the characters in the series, animators, TV-show writers, and voice actors will go out of business. Who needs professional artists when an AI can generate 9 Da Vinci-quality pieces of art in 10 seconds? Or professional singers if an AI can write and record a song in a few seconds/minutes?
Sure, people will still do all of those professions as hobbies, but it will probably be much more difficult, if not outright impossible, to make a living off of doing them. Hopefully, AI will be used to improve the average human life and give us more free time, but I can't imagine corporations and billionaires not trying to monetize the technology and use it to make the world worse.