First, I assume you aren't planning a Stone Age campaign--you want more or less the standard medieval tech level, but with very low population and mostly isolated settlements. In that case, it makes sense to have your setting be one which used to have large civilizations, but suffered some sort of crisis. Medieval technology represents thousands of years of exchange of ideas across large networks of people. A setting where such networks have never existed is never going to get much beyond basic agriculture--indeed, it's unlikely to get even that far. As a bonus, living in a world of fallen civilizations provides a handy explanation for why there are troves of magic items and treasure buried in ruins across the land.
So, assume a world in the wake of some deadly crisis that wiped out most of the population. The first thing is to figure out what the crisis was and how long it's been. Was it a recent event, and the world can be expected to start recovering in a few decades? Or was it hundreds of years in the past, and there has been no recovery and none expected? I suspect "hundreds of years in the past, no recovery expected" is more the feel you're aiming for, in which case the question of why there has been no recovery is important. People are resilient. Give us a chance, we bounce back. There has to be something preventing that recovery. In some sense, the crisis must still be going on. Perhaps a plague of demons or monsters was unleashed way back in the day, and the monsters continue to haunt the wilderness between settlements, making trade a dangerous endeavor and preventing any settlement from growing beyond what its walls can defend.
Next, consider the implications on how the game works. Forget magic item shops--in the absence of trade networks, even mundane gear will be hard to find. Let's say the party walks into town with a nice haul of gold from its last adventure, and wants to buy full plate for the fighter. Oops! No one here can make full plate. There's no iron mine in this town; and with minimal trade, they have no way to get iron from elsewhere. The town's blacksmith operates by reforging scraps and broken tools, or else they've stopped blacksmithing altogether and rely on tools made out of wood, bone, and flint. Making something like full plate is utterly beyond them. You'll have to go on a quest to find a town with a working iron mine, or (more likely) find some leftover armor from days of yore that can be reforged and refitted.
I've written some stories set in worlds like this. They're a lot of fun to explore, but you do have to rethink a lot of assumptions.