DISCLAIMER: I think that different editions fit different playing styles and different people's tastes. What I'm looking for is advice concerning my tastes and playing style. Please do not start edition wars here - if an edition doesn't fit my playing style it doesn't mean that it doesn't fit other people's playing styles.
While I'm currently playing 3.0E, I'm having more and more problems with this system, and I'm looking into the possibility of switching to another edition. The thing is that I'm not sure about which edition will be best for me and my players.
What I'm looking for is:
I voted 2E, and I'll explain why at the end. But first, We'll look at things individually:
1) I like to play fast, rules-light, with as little book referencing during gameplay as possible, and with some level of improvisation. I like simple game mechanics that I could easily learn by heart and thus avoid excessive page-thumbing during the game.
For this you are looking at 2E, basic D&D, or 4E. 1E while somewhat rules-light ends up requiring a lot of book referencing in the end. 3E is terrible for what you describe. 2E and basic D&D perfectly fit what you describe, and while 4E can get complex with combat and character creation, after a short period of time you stop referencing the books altogether, and it can easily do improvisation in both combat and non-combat situations, particularly outside of combat.
2) I like to best utilize my prep time, which is becoming more and more limited as I grow up.
4E all the way on this one. 1E/2E aren't too bad with this though, and 3E is a disaster.
3) I usually DM for a relatively small group of 1-3 players.
1E/2E does a small group best. 4E really wants at least 4 PCs, though I could see things working with 3. With that small of a group, I'd recommend a DMNPC or allowing people multiple characters. 1E/2E does running multiple characters better than newer editions, and multiclassing helps tremendously. 3E does not handle players running multiple characters well(less well than 4E, and the game can be swingy to the point where running 2-3 characters gets very risky. On the other hand, with heavy optimization and powergaming a small 3E group will do well.
4) I usually play Core-centric games with minimal use of additional books.
I'm not a big fan of 3E core. Its the least balanced version of D&D in Core, and a lot of primary concepts just don't work well(Fighter, I'm looking at you). 1E, 2E, and 4E work just fine out of the core books.
5) My settings usually have a low-to-moderate magic level.
I think 4E is very underrated in this regard. While players can spam magic end over end, the magic they can wield isn't high impact like earlier editions. Slow leveling 1E/2E with a cap at around level 10(how most people have historically played it) and frugality with magical treasure can accomplish this goal as well. 3E doesn't do this well at all without heavy modification.
6) I like to use magic, character powers, and monster powers both in and out of combat. Me and my players also like cool "non-combat" spells/powers that could be used in creative ways both outside and in combat.
Actually, 3E does this better than any other D&D. The nature and limits of 1E/2E really require/encourage you to spam Fireballs and whatnot, to the point where its often hard to justify memorizing Fly. 4E's Rituals work very well for "non-combat" situations, but their slow(10 minute) casting doesn't allow for tactical non-combat application, or use in combat at all.
7) I don't mind some boardgame elements in my games, as well as moderately complex combat (up to and including some use counters/minis), but I don't think I'll go to an extreme in this part of the game (that is, I don't think I'll like using extremely detailed tactical gameplay).
What you say here doesn't sound like it disqualifies 4E. I would call 4E firmly in the moderately complex combat camp. Any edition sounds like it would work fine.
8) I'm not very concerned with game balance, as long as I could create (or even eyeball) more or less fair encounters. I (and my players) also prefer to let the characters grow organically rather than be planned ahead. There is also an element of "casual gaming" (read: bothersome RL constrains) that means that we'll be unlikely to reach truly high levels of rule mastery.
This all but disqualifies 3E, particularly the difficulty in eyeballing encounters. The other editions all work fine.
9) I like varied, weird and interesting classes, races and monsters. I don't mind really weird ones. I also like to have some freedom in race/class combinations.
Basic D&D just doesn't deliver this at all. 2E(and 1E if you ignore racial restrictions and level limits), is underrated in this regard, as it offers things like Fighter/Mage, Fighter/Thief, Mage/Thief, ect. The multiclassing system adds a lot of options. 3E offers the most freedom, but the system tends to break down quickly if you aren't optimizing. 4E in this regard is underrated, having limited but deep choices in core.
10) I like a certain degree of world-building, but in D&D it doesn't have to be a very accurate simulation of reality.
3E wouldn't work well in this regard given your time availability limits. The 2E DMG probably has the best tools in this regard, while 4E advocates starting small and fleshing out a greater area as the game progresses.
11) I'm slowly moving from using die rolls for various dungeon activities (i.e. searching for traps/secrets, dealing with puzzles etc) towards a more narrative style of DMing.
2E handles the narrative style best of all editions, and the 4E skill system can easily be disregarded to achieve the same effect. With 3E, you'll be ignoring large portions of the game, and 1E had lots of fiddly dice systems for this sort of thing.
2) I like both dungeons, wilderness exploration, and social/political role-playing.
All versions are equally do well for this.
So, which edition of D&D do you think would best fit my preferences?
As I said before, I recommend 2E. I believe you can accomplish what you want with 4E, but 2E is more what you want right out of the box.