Near to the inn are the butcher's place (shop underneath, cellar for the storing of meat, and living quarters above.) After that there's a fish market, a fruit store, then a vegetable store. The inn happens to be near the part of the village where the village housewives do their daily shopping for groceries.
A few doors beyond that and you get to slightly different types of places--the blacksmith, the local church, the village hall, and then residences of farmers or fisherfolk who leave to work the fields or their boats during the day.
The three men are still approaching the front door to the inn, although they will arrive within a round or two. One of them is also holding a bag that clinks as he walks. All of them have bared cutlasses, and black-powder pistols tucked into their belts. They don't look like villagers--they look more like mercenaries or merchant marines.
Other than them, nobody is nearby in the street or behind the houses. You can hear the bustle of villagers wrapping up their business for the day, but they are further away and out of sight for the time being.
Feo, for the first building or three you'll find either no back door, or a heavily locked one. You can either continue looking, attempt to pick a lock, or take some other course of action.