After choosing your RPG system of choice (you seem to be going for 4Ed D&D):
1) Buy the GURPS mysteries sourcebook mentioned above. I hate GURPS and haven't seen that particular book, but GURPS books are generally well done and researched and tend to be chock full of solid advice and yoinkable goodness.
2) Consider
what kind of mysteries you want your PCs to solve. There's lots of various sources to consider- the L&O or CSI TV series' give you all kinds of murders and other crimes to solve, as do classic TV shows like Columbo, Banacek, and so forth. OTOH, there are also shows like Ghost Hunters, Friday the 13th: the series, X-Files, the Scooby Doo shows, and comics like Challengers of the Unknown in which the goal is seeking to uncover the truth behind paranormal events. Then there's pulp adventure mysteries as popularized by things like the Jon Carter, Doc Savage, Indiana Jones, National Treasure and Quatermass franchises, or more modern versions like the A-Team and Charlie's Angels.
3) Decide on your campaign's overall tone- serious like Sherlock Holmes, Hercule Poirot or Benjamin January, or with a lighter touch like Nero Wolfe & Archie- esp. in the fantasy reinterpretation in Glen Cook's Garrett & "The Dead Man." Even nearly pure comedy can work, like in Moonlighting or The Venture Bros.
4) Set PC age. It matters somewhat- are your PCs adults, or are they younger "detectives," like Nancy Drew, Encyclopedia Brown, Light Yagami, L, Near, Johnny Quest and the Hardy Boys...or even an amalgam like Jimmy Kudo?
Kids can sometimes get away with things adults can't, and are often underestimated by their elders. Meanwhile, adults can usually go places no child can go and have connections and licenses no child can match. This dichotomy matters more in modern society, but would still be present in a typical FRPG setting.
(Check out post#146 in this thread for the Nancy Drow Mysteries. There are also a bunch of other good ideas in there...
http://www.enworld.org/showthread.php?t=102706 )
5) Don't overlook the current crop of modern fantasy. Many of the characters within tend to be mystical detectives of a sort, like Simon Green's Nightside and Jim Butcher's Dresden Files novels.