If you're looking for a secret handshake, you can't beat the d20. Every D&D player, from the white book-toting, Gygax-quoting grognard to the 4E dragonborn warlord enthusiast, will recognize a d20 on sight and associate it with D&D. Conversely, non-gamers are unlikely to know what a d20 is, especially if you leave off the numbers so it looks like an abstract symbol.