Would you even require a system for everything besides the combat you do to get the exact same experience and support for it?
Yes, a system was definetely needed. I'll expand about what I wrote above.
In a totally freeform system, action resolution would be "hey, I am a knight, I grew up with nothing to do besides jousting with other knights, shouldn't I be able to recognize this coat of arm if it's from somewhere close?" and the DM would say yes or no. Here, we're trying to mobilize two gaming elements: the raw power of the character (stat bonus) pertaining to the task & a reason to claim proficiency. The game aspect is important even if it's light in our group. Without it, we wouldn't have the impression of /playing/, we'd just have fun talking. Hence, we adopted all the rules that allowed for farming for proficiency (background and tool, in a non-limitative way) and we use skills with any stats when appropriate. Refereeing a boxing match is WIS+athletics.
The enabler of a fight-llight game are also in the hand of the DM. Only he decides that a fight happen. Most of the opposition can react to officers of the Watch like you or me in the real life react when a police officer pulls us over: minimal manslaughter is expected. Playing in an urban settings help: the things the opposition cares for would be lost and killing a single Watchman won't solve the problem ; on the other hand, the players are beholden to not kill citizens unless there is very much no other choices. It doesn't mean you can't engage with NPCs, including hostile NPCs. Action scenes can happen without ending in a fight, even (hence a house rule adaptation of skill challenge, that I regretted seeing dropped after 4e). It makes fight, where the tension escalate to life-threatening proportion, all the more important (and I try to make them memorable, which I think I couldn't do with the number of encounters per day I'd have to design with the default ruleset).
Even doing published adventures doesn't require
much change to be adapted to something my group will like.
[TBH, with regard with magic being rule light as well, I also regret the Incantations rules from 3.5 were never refined into a full-fledged system]