I can answer this, my company motto has the word vanity in it.
Seriously, before the "startup" cost at RPGNow of $40, you probably will want to register some form of legal enterprise depending on the laws where you live. At a minimum file a DBA (doing business as) with your county clerk's office. This can be cheap and it can be expensive depending on how elaborate a company you wish to form and how much your local officials rip off startup companies. Again depending on how elaborate a company you are forming you might get away with talking to a tax accountant (if you use one) in lieu of an actual attorney. Or at least if you know an accountant he might recommend a lawyer.
On the legal front, if your work involves the d20 and OGL licenses, make sure you know how they work and/or consult an attorney in your jurisdiction.
The next biggest (and ongoing) money outlay is a company website. This is also a time sink to go along with the marketting, advertising, etc expenses that detract from time writing.
Getting known requires getting something out there and getting people to know about it. This is where most businesses fail. If you are truly a vanity press, you don't care about this.
Me, I don't mind wearing a lot of hats. I also don't produce a lot of material. (Heck, I'm mostly coasting on good word of mouth from my first releases. You can't really do that since I did it when there were far fewer vanity d20 PDF publishers around and you could release an unillustrated PDF and still get good reviews.) Although I am about to release something written by someone else. A first for me.
Finally, you should read these boards, industry boards at rpg.net, and the publisher boards at rpgnow and familiarize yourself with the industry. (Another time sink.)
Oh, and if you go through with this after all. Hire an editor. Don't do it yourself. Don't hand it to a friends to read over. Hire an editor. Do not scrimp on this expense. There is nothing worse than good material buried under bad editing. And first impressions do matter.
And if all that is too much work, submit a proposal to an existing publisher. Visit websites to find out which publishers accept such proposals.
Good luck.