Having lived in Japan and being married to a Japanese woman, I can perhaps bring a little perspective.
Animation in general is more accepted as a form of entertainment for both kids and adults in Japan. Whereas in America, animation is pretty much considered kids fare.
However, I would still say that 65% of Japanese culture considers anime fans just as nerdy and socially outcast as probably 85% of Americans do.
That being said, you can't really lump all anime together. Sure, those who don't like it and aren't too familiar with it can do that, but anime is pretty diverse as a form of entertainment.
There is anime for children (Pokemon, Yu-Gi-Oh, etc.), there is anime for families ala Disney (Kiki's Delivery Service, My neighbor Totoro), there is porn anime, there is over-the-top insane super-power wielding anime (Dragonball Z, Fist of the North Star), cyberpunk (Ghost in the Shell, Akira), and so on.
I'm not a big anime fan, but some of it is rather good, IMO. Anything by Miyazaki Hayao is on par with or exceeds Disney in terms of thought-provoking stories and family entertainment. Ninja Scroll is cool as a D&D esque Asian fantasy flick that takes place in a mystical Japan. Fist of the North Star is basically a story of Justice League level superhumans in a blasted post-apocalyptic wasteland. Think Road Warrior with Superman. No its not Shakespeare. But its a killer beer and pizza movie to watch with your friends. Even you watch it just to make fun of its over-the-top superpowered heroes. And the quotes! Man, I haven't see the movie in years but we still quote it at the game table!
Macross Plus is an excellent Top Gun style story set in a future sci-fi universe. It also has some interesting themes dealing things like man vs. machine, the nature of love, and artificial intelligence.
Some anime is really hard to understand unless you know the backstory. Akira, for example, is based on something like a 2000 page manga series. All of that condensed into a 2-hour movie. If you don't know the manga, the movie will probably be pretty opaque to you.
Anime excels at the cyberpunk genre. I'm not a big fan of cyberpunk because I don't buy the evil corporation dystopia nonsense. I don't really care for anime such as Ghost in the Shell. Although, even I have to admit its a very cool movie as far as cyberpunk goes.
So even if you think you hate anime, I'm sure that somewhere is an anime that you might like or at least tolerate simply because its so broad and diverse.
But some anime is great to watch in a MST3K sense with your friends. We watch Dragonball Z sometimes just to laugh at the bad guy who spends 10 minutes charging up enough chi energy to nuke a planet from orbit, and then get his butt kicked by the little kid with the spiky hair.