Carnies... Circus folk, nomads, you know. Smell like cabbage. Small hands.
Okay, obligatory joke aside, Philip hit the nail on head. The scariest thing is the unknown. From that sound you think you just heard from over there in the dark, to the fact that you just entered a town that should be teaming with people but appears abandoned. The worst part of any horror is simply being caught up in something you don't understand and can't explain.
For myself, I've got pretty active imagination, so it doesn't take a whole lot of anything to set my skin crawling. Seeing a movie like The Ring sets my "scare the crap out of me" section of my brain on overdrive. Ditto with certain books (Stephen King's The Tommyknockers).
Heck, I don't even need a catalyst. Set me in a house, by myself, on a quiet night with nothing to do, and watch the fun. Thankfully, I've got decent self control and a firm grasp on reality, otherwise I'd probably end up bugging the crap out of a friend everytime the electric went off, lol.