I can thing of at least four fantasy or fantasy-like subgenres that fit these criteria. Unfortunately, the widespread influence of D&D has choked out most of these options in terms of published RPGs. A system like Conan the Roleplaying Game, d20 Modern/Past or Grim Tales could handle them in d20, though.
Alternate history.
Sci-fi with a fantasy gloss hiding the technological backstory.
Supernatural (but not magical) horror ala vampires, ghosts and werewolves.
A world without any magical effects but with different history and geography from ours.