The art was a distraction. Even the more in-depth images (the halfling fighter for example) were more distractions than useful illustrations.
The book had too many game mechanics. Character skill in sex used far too many resources to ever allow an adventurer to really be able to invest in it and survive adventuring, despite the high likelihood of being experienced. As mentioned, some of the mechanics were simply silly, unnecessary, or just made me ask 'why?'. On the other hand, it had a lot of good, interesting, or well thought-out mechanics, showing the skill and insight of the design and development team.
A prime example is the creature section. While I often asked "why did they make this creature? why didn't they make this other creature?", the creatures themselves were (mostly) well designed. Oddly enough, much of that chapter focused upon creatures that are already in D&D, and mostly in the first Monster Manual (half fiends, half celestial, succubus, half fey, giant kin, etc.), making that section utterly superfluous.
There wasn't nearly enough advice on how to incorporate sex into a fantasy game tastefully. Like many other gamers, I'm often unintentionally offensive, rude, or awkward. RPing sex is going to have plenty of moments of all three, without the addition of the raw talent so many of us have. Advice on how to incorporate it smoothly and tastefully, or explicitly yet inoffensively for those groups that want that, would have been very much worthwhile.
The 100 adventure ideas what a good start on how to incorporate sex into a campaign, but it was just slapped onto the pages with no further development. Likewise, the organizations were interesting but I couldn't use any of them in any setting I play, with a straight face. No good at all.
Overall, the Book of Erotic Fantasy aimed to be the definitive d20 book on sex in fantasy gaming. The only reason it's succeeded is for lack of serious competition.