I second Klaus on "The Festival" and the "Lurking Fear." A Lot of HPL's older stuff is quite good and in some ways even more unique than the more mytho-sy stuff. I read a big chunk of it a couple months ago and enjoyed it thoroughly.
I also strongly recomend a story called "The White Ship." I'm currently working on and off on a rather stream-of-conciousness type story losely inspired by that one. It has an almost poetic feel.
The Colour out of Space was the first of his stories that I read. Call of Cthulhu, The Dunwhich Horror, Shadow over Innsmouth, Pickman's Model, The Dreams in the Witch House, all of those are great. The Dream Quest of Unknown Kadath is wonderful to.
As to HP's skills, and why I love him like I do...well theres a lot of reasons. A big one is the simple fact that he's different. His writting style is very unique...his creatures and beings also are strange and different not your usual by-the-mold beasties.
Also, especially his older stuff has just a wonderful dream like quality...not neccesarily much of a plot, just ideas and concepts and images.
Really, I love Lovecraft for many of the same reasons I love Anime.
While Joshua is right that he did consider himself a "horror" writer, his idea of horror is/was a bit different than many people's. It was less about imediate fear, about being scary, and more about a sense of "cosmic dread", the idea that the world is a far less inviting place that we had thought, and that if we looked just a little ways beyond our day to day sphere we'd encounter things that would leave us unhinged and afraid for the rest of our lives. I think he usualy succeeds at presenting that kind of fear, personally.