To a great extent it depends on the definition of "mature." The 3E Book of Vile Darkness, for example, claimed to be mature content but was in fact quite the opposite; a lot of immature silliness where "vile darkness" appears to mean BDSM and topless women with lots of piercings. (Magic nipple clamps? Really?)
There are two ways you could go with a Book of Vile Darkness or similar sourcebook. If you wanted it to be really mature content, you'd portray evil in a realistic way and pull no punches. Such a book would be aimed purely at the DM, to create truly vile opponents; it would be a twisted player indeed who could read about the real-life effects of murder, torture, rape, genocide, and so forth and then set out to create a character who did those things.
I wouldn't buy such a book, though, and I don't know that I'd even want to see it published. You're venturing into real-life territory there, and there's no way to know if someone in your group is a survivor. (Yes, even of genocide. It still happens.) You could seriously hurt people and wreck friendships by introducing that kind of material. I have a hard time imagining a use for such a BoVD that outweighs the risks.
So my preference would be to go with a cinematic portrayal of evil and dump the "mature content" label (unless you're worried about reviving the D&D-is-Satanism craze). Devote chapters to various types of cinematic villainy and include examples, mechanics, and advice for creating memorable villains. This sort of BoVD could have some player crunch for the occasional evil party, perhaps even a chapter or two devoted specifically to evil PCs. I'd buy that if it was well written.
As for a Book of Erotic Fantasy? Well, I guess the question there is whether the goal is to take a typical RPG campaign and add a few risque elements, or to make the RPG equivalent of erotica/porn--a campaign revolving around sexual situations and activities.
If the former, you'd probably want to focus on mechanics and plot techniques for incorporating sex and love:
- An introduction laying out the types of things you might want to incorporate and how to approach them (are you looking for tragic romances, or ale and whores?).
- Guidelines for developing romances and playing out seductions, with the option to "draw the curtain" at various points depending on how explicit you want to be.
- A chapter on possible "side effects" of sex, like STDs and pregnancy, with mechanics designed to discourage DMs from using these things as "gotchas" for players.
- Some material on sexual mores in real-world cultures both past and present.
- Equipment and hirelings (yes, that kind of equipment and hirelings, but also stuff like fertility drugs and cosmetics).
- And, of course, the chapter on how to establish what level of explicitness is and is not acceptable in your gaming group.
Eric's grandmother is watching, so I'm not going to go into what the other type of BoEF would entail. Let's just say I don't think you'd have much of a market among D&D's standard player base; I have never been in a gaming group where I'd feel remotely comfortable with a campaign like that. On the other hand, you might be able to pull in quite a few new players by advertising in... other venues.
