I am also in the camp of "rare magic", for several reasons:
1. I am a lazy DM. I don't want to spend hours preparing a session just to properly challenge characters numerous magic items, spells and abilities and to make sure I don't overlook some easy way out of a problem. I don't want to trick out every opponent with magic just to make them competive. Having magic rare in my campaigns and certain spells banned means it is much easier for me to prepare an adventure, and that I can focus on the plot and NPCs, not on the stats.
2. I am greedy DM. I want to have as many adventure possibilities and challenges at my disposal as possible - I don't want to be forced to either drop staple adventures and challenges (f.e. travel to X, climb over the wall) or to use the "plot preservation nerf device" (teleport won't work in this case, there is an anti magic aura around the wall) excessively. I don't want to have to resort to "standard tactics" each fight or assault (like dispel magic on buffed party, amulet of nondetetction for spies). Having magic be rare in my campaigns makes it easier to vary the challenges and events. I can have the group captured more easily, present them with more challenges they can't just solve with a wave of their hand or wand, and don't have to back up any kingdom with tons of spellcasters just to keep it intact.
3. I am a cinematic DM. I know how I want my campaign to look and feel like, I know what kind of battles I want. I don't want "MagicTech" land, where you just replace science with magic, and have a modern feel to the place. I don't want the PCs to be the sole "magicTech" users either, which tons of magic stuff at their disposal, battling equally equipped small numbers of foes. That reminds me far too much of superhero comics, where there are cyborgs and space ships for the elite few but the world at large remains unchanged, with people, even important ones who could afford it, remaining crippled, and the population having our tech level, even the military and police mostly.
I want a campaign world where magic, at least higher magic, is rare, yet not domineering. Where PCs can be heroes wielding a few magic items with rich histories, but are defined through their skills and abilities, not their gear. A world where castles make sense, since flying mounts or monsters are not common and teleport very rare. A world where you don't ask yourself why the nobles are not replaced by wizards, or why the PCs don't take over the town by force, or why they don't just simply raise dead the unintended victims of their latest fight.
I don't have my PCs fret about cost of living and such, or hunt rats to pay the rent. In fact, I drop such costs - in my current campaign, the PCs are living in luxury, and we don't count gold or such. But I do not think they need teleport, scry, 100s of spells and items and every PrC in the books to influence the politics of a kingdom or fight demon lords. When I reduce the amount of options and counters (drop the DR instead of giving everyone +X weapons, drop the number of spells around instead of giving each opponent some dispel magic lackey, drop DCs instead of adding protective items and so on) the end result is the same - a vast plot or campaign arc, or a close fight.
My Forgotten Realms are heavily modified to fit this, and - IMHO - it works out well.