I'm going to buck the trend here. Personally, I'd definitely encourage you to look at OSRIC, C&C, and 4e.
With that said, I'd like to encourage you to look at Warhammer Fantasy Roleplay, 2e.
1) I like to play fast, rules-light, with as little book referencing during gameplay as possible, and with some level of improvisation. I like simple game mechanics that I could easily learn by heart and thus avoid excessive page-thumbing during the game.
Yes, absolutely. You will seldom need to reference the book, except maybe during prep time to figure out what the heck a certain Trait is.
2) I like to best utilize my prep time, which is becoming more and more limited as I grow up.
Prep is pretty simple, I'd say, but there are literally dozens of free adventures available online.
3) I usually DM for a relatively small group of 1-3 players.
It'll work fine for this.
4) I usually play Core-centric games with minimal use of additional books.
You can get a very satisfying WFRP game with the core book and Old World Bestiary. Everything else is expansion. I've bought a lot of them, and if I were to recommend other books, they'd be Tome of Salvation, Tome of Corruption, and ... the magic book, whatever it's called.
Still, none of these are anywhere near essential.
5) My settings usually have a low-to-moderate magic level.
Heck yeah. Magic is dangerous. Magic items exist, but are very, very rare. And, many are tainted by Chaos!
6) I like to use magic, character powers, and monster powers both in and out of combat. Me and my players also like cool "non-combat" spells/powers that could be used in creative ways both outside and in combat.
Oh, absolutely. By definition, most skills and traits have non-combat applications. Spells, too, if you happen to have a caster around.
7) I don't mind some boardgame elements in my games, as well as moderately complex combat (up to and including some use counters/minis), but I don't think I'll go to an extreme in this part of the game (that is, I don't think I'll like using extremely detailed tactical gameplay).
It's as board-gamey as you want to make it. You can run combats with minis & grids, or you can run combats just in mind-space with few difficulties.
8) I'm not very concerned with game balance, as long as I could create (or even eyeball) more or less fair encounters. I (and my players) also prefer to let the characters grow organically rather than be planned ahead. There is also an element of "casual gaming" (read: bothersome RL constrains) that means that we'll be unlikely to reach truly high levels of rule mastery.
You're fine here. It's easy to figure out how tough something is by looking at its Weapon Skill, damage, toughness armor, and wounds.
9) I like varied, weird and interesting classes, races and monsters. I don't mind really weird ones. I also like to have some freedom in race/class combinations.
Weird and interesting classes? How's Rat Catchers, Vagabonds, Troll Slayers, Hedge Wizards, Grave Robbers, etc. strike you? Seriously, there's 30+ Basic Careers, and a similar number of Advanced Careers.
On the other hand, there's only 4 races, unless you go insane and run a Skaven or Chaos campaign. The combinations are pretty freeform, though.
10) I like a certain degree of world-building, but in D&D it doesn't have to be a very accurate simulation of reality.
Well, WFRP2 has the Empire, which is pretty darn comprehensive. IMHO, it's quite tied to the setting's default assumptions, which may turn you off. I've never been interested in trying to re-skin the setting, so I can't say how tough it'd seem...
11) I'm slowly moving from using die rolls for various dungeon activities (i.e. searching for traps/secrets, dealing with puzzles etc) towards a more narrative style of DMing.
There's skills for investigation, and it's less free-form than 1e/2e. Still, you should do just fine with narrative exploration.
12) I like both dungeons, wilderness exploration, and social/political role-playing.
Yes, yes, yes. WFRP2 is actually
very well-suited to low-combat games, with its robust skill & talent system.
-O
