Guards should be at least as powerful as the opposition they're supposed to guard against on a day-to-day basis; otherwise the opposition, rather than the guards, would control the town.
In most cases that means facing various CR 1/6 to CR 2 creatures such as goblins, kobolds, orcs, hobgoblins, low-level thieves (not necessarily Rogues, more typically commoners or experts) or bandits, and the occasional bugbear, ogre or higher-level/PC-class rogue/bandit. Higher CR monsters would either be rare, reside in areas far away from civilization or spend most of their times in ruins and dungeons. So Warriors of levels 1-3 with decent equipment (chainmail, large shield, longsword or short bow or light crossbow) would do well. Larger cities would probably have some higher-level veterans or specialists (including spellcasters) as well a SWAT equivalent (mid-level PC-class elite unit with good gear used to deal with major threats).