Modification of the spell component rules is an option. In my experience, most DMs and players only worry about expensive or rare spell components. If you wanted to make magic more difficult or rare, make the components required for spellcasting more rare.
One thing that I'm not so thrilled...