Uller
Adventurer
Well, different strokes and all that - I love town encounters, and my players kind of do, too. I run a sandboxy game, too, where I always drop three hooks to the players - each one is a different adventure. The players pick which adventure they want to "bite", and I prep it. It's working pretty good so far.
I don't think you and I are all that different. What I'm talking about getting away from is the detailed map with numbered buildings each with 1-3 "named" NPCs that hold some piece of information or something else of interest to the PCs resulting in spending a session or so "exploring" the town like a dungeon crawl. Red Larch in Princes of the Apocalypse and Phandalin in Lost Mines are both like this and both have resulted in some boredom in each of the 3 times I've played them (Red Larch and Phandalin as a DM and Phandalin as a player).
What I'm looking for is how other people avoid this. I sounds like you avoid it pretty well. I like Isreth's method because it gives the players a chance to input their imagination into the town. My daughter is one of my players. She is playing a gnome arcane trickster and is constantly looking for ways to cause mischief. I often have to tone her down to avoid her taking too much spot light time from the group. This way, she can say what kind of trouble she caused and how she got out of it (or I can make her make a check and impose some consequences). If one of my players wants to scour the town for hooks, rumors or whatever, he can do that without taking three hours of table time while I struggle to make important NPCs that have one minor piece of information interesting.