There are several reasons:
First, Sony tried to make the PSP a jack-of-all-trades, which made it a master of none, as TwistedBishop noted.
Second, Sony tried to tie it to UMD, which was an utter failure, because who wants to buy a movie only watchable on PSP? That was a very poor choice.
Third, it comes with a tiny Memory Stick, and high-capacity Memory Sticks are more expensive than, say, SD.
Fourth, it has poor format support. It's kind of a pain to rip videos to it.
Fifth, Sony keeps trying to prevent homebrew apps from working with firmware updates. While there are somewhat valid reasons for wanting to do so, letting the community do cool stuff with your hardware increases its value.
Sixth, it doesn't have as many great games as the DS. If it did, it would be more popular and thus, I think more people would use it as a media player, even if it's far from the ideal portable media player.
Seventh, the big screen is nice, but the system is bulky. Plus, when I've tried to put it in my pocket and use it as a music player, its buttons often hit the side of my pocket and stop the music.