Obryn
Hero
You can play an excellent and complete 4e game using only the three books I'd call "true core." That is, the PHB, MM, and DMG.
If you want to start small, that's a good place to start!
From there, I'd probably add sources in more or less this order...
* Any DM-only book you care to add. No need to make up your mind here.
* Adventurer's Vault (mostly for the regular equipment, including superior weapons)
* PHB2 (Races, classes, and feats! OH MY!)
* FRPG (mostly for Drow, Genasi, and Swordmage)
* Eberron PG (mostly for races + Artificer)
* Martial Power and/or Arcane Power
* Dragon Magazine
I'm putting the Power books near-last because I think it's usually better to get more breadth than more depth.
While I think they're excellent - particularly Arcane Power - they're intensely optional.
I have almost no concerns about the balance of Dragon articles, but lots of the stuff in them is pretty niche (as IMO it should be!) While they're neat additions, you could probably play for years and never miss them.
-O
If you want to start small, that's a good place to start!
From there, I'd probably add sources in more or less this order...
* Any DM-only book you care to add. No need to make up your mind here.
* Adventurer's Vault (mostly for the regular equipment, including superior weapons)
* PHB2 (Races, classes, and feats! OH MY!)
* FRPG (mostly for Drow, Genasi, and Swordmage)
* Eberron PG (mostly for races + Artificer)
* Martial Power and/or Arcane Power
* Dragon Magazine
I'm putting the Power books near-last because I think it's usually better to get more breadth than more depth.

I have almost no concerns about the balance of Dragon articles, but lots of the stuff in them is pretty niche (as IMO it should be!) While they're neat additions, you could probably play for years and never miss them.
-O