If you want to keep spellcasters rare without diluting the power of the individual too much, require all casters to spend a Feat to "unlock" their magical power. For example, the Arcane Affinity feat might require INT 15+, give the Wizard's cantrip abilities, and allow access to the Wizard class. The Divine Affinity might require WIS 15+, give the Cleric's domain power, and allow access to the Cleric class. And so on.
If you want magic items to be rare, there are all sorts of economic/social/political things you can use. A healthy anti-magic paranoia goes a long way for this. Having more small towns and fewer large cities helps when you consider the DMG pricing guidelines.
Or, just double the cost of all magic items (both GP and XP costs).