It was a tough call between 3rd and 4th (I went with 4th). In my world, casters aren't necessarily a rare thing in certain places. There are large academies of arcane magic, a continent-wide guild with halls/towers for use (by those willing to pay the dues) in large population centers, temples of the gods with various orders and followers spread across the world - from lone shrine-keepers to cloistered orders (abbeys, monasteries, priories, etc...) to cathedrals and massive basilica.
I will note here that not all members of a religious order or priestly hierarchy are full spell-wielding "clerics." Though many in the upper eschellons are. But all clerics are attached to some deity's religion/order.
There is a continent-wide druid organization keeping watch of the esoteric Balance and the affairs of Men. There are various organizations (some secret, some not-so) of magic-workers of different kinds, master mages tutoring apprentices in the "old style" of teaching/passing on magic study, researchers in a mage-ruled realm -alchemists, magic item makers, new spell creators, etc..., and, of course, player characters and established world NPCs.
So, magic USE isn't necessarily all that uncommon.
There are the typical ages upon ages of empires, realms, and peoples that have risen and fallen, some more interested in magic than others, some very specific in their areas of expertise (I have an old nation of serpent-demon worshipers, who were bio-magically altered and experts with magically imbuing living creatures and plants, fungi for their purposes. Basilisks, medusae, and other creatures remain in the world because of them. Another "lost/fallen/ancient empire" created massive Colossal constructs. Any hope of producing anything beyond a basic golem or simple animated suit of armor, like say a gargoyle, you'd better find some research, writings and materials, from that ancient land. And so on.
So there are many items from the world's past. There is a slate of known items from various points of history: certain weapons, magic swords, staffs of great wizards, and the like, that might be in the possession of some known person and some that aren't or "got lost." Gnomes are known to produce a good deal of potions, some elvish items, and that mage-ruled wizard-nation produce some simple items or minor enchanted weapons in which they trade.
Magical scrolls are pretty common place. Potions can be found in most medium or larger settlements...the odd traveling salesman or lone alchemist conducting their research. But you're not going to find a "wand of your choice" or random "chachka of useful magic" in the general corner store or a flaming sword +3 that speaks five languages that just so happens to be in the farming village you strolled into.
Things are available or you can go out and try to find them. Flat out buying something or paying a wizard to enchant items for you is going to be difficult to come by anywhere other than major metropolitan areas, and isn't going to be cheap.
In the world, you're going to find casters. People will use what utilitarian magic they find access to. But, if you think about it, even someone who can turn invisible or entrap people in conjured webs (both only 2nd level spells) is pretty effin' powerful among the "common folk." Throwing around fireballs and dispelling other mage's magic is the "real power" level that most non-adventuring mage or priest is going to attain.
So, that's why I went with 4th. You're going to need to find a really ambitious or very accomplished, like someone who would be administrating a guild tower or teaching at an arcane academy, or a remarkable hero (or villain) to be reaching into that level. Once you get into 5th level spells, you're on the "world altering" level - teleporting, raising dead (not just "animating corpses"), it's BIG stuff. So, I'm saying "4th" is where "real impressive power" is going to start in the campaign world.