I mean, it's a feedback loop. The swastika was a generic good-luck symbol before 1933. Skulls are a symbol of death and therefore look dangerous and menacing, so Fascists used them to look hardcore like bikers after them.
There's a whole hypermasculine look to fascist iconography people can be drawn to without actually being fascists. I mean, you might fantasize about being in an army and killing the faceless enemy without actually wanting to do it in real life. People watch heist movies and fantasize about outsmarting the law and escaping with a bunch of cash, but most of those people don't go on to rob banks.
Nazis are powerful and evil, which is attractive to many as well (look at Dracula or Gordon Gekko or even the Joker in some cases)--apparently in the BDSM community there's a whole ongoing controversy on how to get the evil 'look' without being offensive to anyone whose family was actually killed. Why don't people get the same feeling about Communists, though? I mean, the NKVD and KGB did lots of bad stuff. I suspect, ironically, the effect of Hollywood (which of course had lots of relatives of the Nazis' real-life victims behind the scenes)--all those movies with Nazis as villains, which of course cast good-looking, charismatic people as Nazis just they like cast good-looking, charismatic people as everything else. Gives them a certain outlaw charisma. Christoph Waltz (Hans Landa, Inglourious Basterds) to take a recent example is a very good-looking guy with a lot of stage presence, and he makes being a Nazi look cool. (The protagonists are too, of course, but when a lot of your villains are Nazis, people wanting to be bad are going to think of Nazis.)