Looking at it from the other side, I've been running a campaign set in the police force of a fantasy city. The police don't need a term for non-adventurers, apart from "people" or "local inhabitants." To them, "adventurer" means a person of unknown disposition and capabilities, possibly sensible, but possibly prone to using their power unwisely. They're treated with caution.