I went looking for my first MP3 player a few months ago. I came here with the same question, and I got some answers, and I am extremely pleased with what I got. So, let's break it down some...
I will assume, from what you've said, that video content and games aren't what you're after. Like me, you're after music. I was also after podcasts and audiobooks.
iPods will do all those. But honestly, the strength of the iPod isn't the actual iPod, but the iTunes store. I am not fond of iTunes as an application to manage your collection, but they do seem to make it easy to buy more music if you want it, and have a large collection to buy from.
iPods are also not particularly cheap. Neither are the Zunes. The Walkman lines is a little pricey, and Walkman dropped some features that supported podcasts and audiobooks, which seemed like a dumb thing to me.
The winner for me was the Sansa Fuze. It handles more audio file types than most players (if you care about not caring what format your music is in), and manages audio books and podcasts well. Gets FM radio, and can take a memory expansion card (and they sell cards with 1000 songs as "slotRadio" for $40). Sound quality is good. I use mine for five miles of walking each workday, and only need to charge it once a week, at most.
The other Sansa players are also supposed to be pretty darned good. Look through them, and pick and choose the features you want.
[ame=http://www.amazon.com/SanDisk-Sansa-Video-Player-Black/dp/B001GCQ3KM/ref=sr_1_1?ie=UTF8&qid=1290452788&sr=8-1]Amazon.com: SanDisk Sansa Fuze 8 GB Video MP3 Player (Black): Electronics[/ame]
I also dislike earbuds, by the way.
Koss makes some good, non-earbud headphones at darned good prices. The Porta-Pro, Sporta-Pro and KSC75's all use the same tech for sound reproduction - the design is decades old, and they keep it for very good reason.
[ame=http://www.amazon.com/Koss-PortaPro-Headphones-with-Case/dp/B00001P4ZH/ref=sr_1_1?ie=UTF8&qid=1290452235&sr=8-1]Amazon.com: Koss PortaPro Headphones with Case: Electronics[/ame]
[ame=http://www.amazon.com/Koss-KSC75-Portable-Stereophone-Headphones/dp/B0006B486K/ref=sr_1_3?ie=UTF8&qid=1290452235&sr=8-3]Amazon.com: Koss KSC75 Portable Stereophone Headphones: Electronics[/ame]
[ame=http://www.amazon.com/Koss-155475-SportaPro-Stereo-Headphones/dp/B00001P505/ref=sr_1_15?ie=UTF8&qid=1290452235&sr=8-15]Amazon.com: Koss SportaPro Stereo Headphones: Electronics[/ame]