folks with allergies to marketing
let's start with a target demographic - 16-20 yr-old male, middle-to-upper income household.
Broad demo - how to refine it? look for gateway activities. Academic engagement - model UN participants, honor society, AV club, Chess club, Quiz Bowl, Robotics club, Anime club? Engagement with literature - book stores, coffee shops, fan clubs, amazon reading list?
I think investment in one or more types of fantastic settings is a key indicator. Comic fans and PnP RPGers have huge overlap because of the investment in the setting. Pern-ites are an exception - and maybe there's an important difference there that I can't figure out. Lots of fan-fic, but no good RPG system, and I've never known a long-running Pern campaign. Tolkien-esque fantasy, Star Wars, Star Trek, Conan, Bladerunner (aka Shadowrun) all have dragged segments of population into RPGs. Vampire/Goth is... it's own thing... but it's definitely got its own voice. A strong desire to engage a setting (or several) is key to RPG'ing, and to identify potential recruits.
Problem solving is also a strong indicator, along with creativity. Part of the reason for the overlap between IT and RPG is exactly this problem-solving aspect. Science fairs, Maker Fairs, Lego Fairs, robotics clubs... just about anyone building something - preferably something fantastic (rather than, say, small engine repair...) is probably a candidate.
How to hook them? This is actually tough - the question of how you cut through all of the more easily accessible alternatives. Honestly, any attempt to read something like EnWorld to get an idea of what RPG playing is like is hopeless (and off-putting). There are no bold visuals (who can compete with skiing and surfing?). There's little performance aspect (ever hung out at someone else's game?). So something like D&D Experience, or gaming conventions, or intro-games at book stores, or something... with an engaging presenter (preferably articulate, young, and curvalicious) is probably the best hook. Second best would be some good form of eperience reports/short-story-based-on-gaming-session or some other first-person account of a gaming experience. I think it might be presentable as a 30-second television-style ad.
I really don't think that standard copy advertising, or click-ads, or anything else that doesn't convey the experience.
How to Retain? People fall out of RPG'ing for two main reasons - their group broke up (or they left it..), or they ran out of time. Out of time is, and will continue to be, an increasing issue. RPGs are definitely a time-consuming hobby. So I think that these two issues are tightly coupled - someone's fallen out because they ran out of time, and someone dropped out because they left their gaming group, and they haven't hunted down a new gaming group. So make it easy for folks to reconnect. A mixer. A promotions party. A new-release event.
On reflection, I have _never_. _ever_. received a mailing, promotion, or invitation from either a FLGS or an RPG publisher. Not one. I've been in this hobby for over 25 years. I _always_ connect with my FLGS, wherever I live. I've playtested for several gaming systems and several publishers. I was an Outrider for Games Workshop (presenter/ambassador). I've had subscriptions to several RPG magazines over the years (I believe they've all folded now...). I attend (and sometimes organize) conventions. I'm not hard to find. I'm not shy. But no commercial RPG entity has ever reached out to me as a customer, and attempted to reconnect me back into the hobby (and I have needed it a couple times).
As a final note - I've seen folks talking about producing continuing streams of good products as a retention tool. Crap. In fact, a stream of good products probably means that good old products are OBE and/or out of print. Meaning that the longer you're out of the hobby, the harder and more expensive it is to re-engage. Longevity of product and investment (intellectual and product) is a retention tool. Churn is not.