If your game could be unraveled by players not having to track their ammo, that sounds like a very fragile and poorly-planned game.
Even D&D left the term referee behind and for a good reason. Referee is necessary when there are two or more sides competeing, collaborative storytelling games require different roles to be taken.
It is neither war nor a sport, and looking at it either way is very limiting.
Because springing that on players unexpected at time of need would be a very crappy thing to do and makes GM look like a douche trying to get one over the players. Even if you have pure intnetions, you are making assumptions about what the characters would or would not do and effectively railroading the players.
An example from my own experience. D&D 3.5, the party saved a tribe of Goblins enslaved by Malar worshippers, they throw a feast in our honor. Surprise, surprise, it's an ambush - tribe's leader figured out our Sorcerer brutally murdered her daughter and sold us out to Malar cultists in exchange for leaving her tribe alone. Malar cultists ambush us and in first round hit with poisoned darts. However, when rolling for attacks, DM asks our AC but tell us to subtract bonus from armor, since we took it off. None of us declared they do, so we protest, and Dm pushes back, claiming "Of course you took off your armor! It's not realistic for anyone to sit down to eat in armor! Only a total sociopath would do that."
Here we have a DM who made an assumption what characters do - take off armor, not refill the quiver - without players confirming it with them. Whenever he meant it or not, such assumption ended up looking like railroading and trying to screw the party over.
I think saying "my character looks for traps all the way in" once is enough, if I'd have to listen to player interrupting the flow to check for traps in every single room, I would go insane.
I'm curious- did most of the goblins also not have their armor on or weapons at the ready?
I've been bothered by tracking ammunition since 4e, when Implement users could use their At-Will ranged powers infinitely, but oh man, if you use a weapon, you better track ammunition!
It's interesting that tracking material components has become so vestigial that in 5e, you can just carry a spell component pouch or a spell focus and basically pretend they don't exist, but the idea of obviating the need to track individual 5 cp arrows is beyond the pale.
The last time I played a dedicated archer, I remember being so annoyed at never being able to recover more than a couple arrows at a time that I eventually bought 100 quivers of arrows and tossed them in a Portable Hole- after a couple of sessions of me saying "oh man, I'm down to 1,700 arrows", the DM finally sighed and told me not to bother tracking them anymore, lol.
But not once has a DM ever asked me about stringing or unstringing my bow, it's all about the ammunition!
A friend of mine is working on his own fantasy TTRPG (I've told him he's insane, but some people just need to learn the hard way, I guess) and he made all these cool thrown weapon abilities that he was really proud of.
He asked me my opinion and I was like "nobody is going to take those, because it's such a pain to carry around a ton of thrown weapons. Then you add in the fact that throwing a weapon at an enemy runs a very strong risk of you never seeing it again. And don't get me started on thrown weapons that are made of special materials or that are enchanted, unless said enchantment causes it to return to your hand."
He frowned at me, but he couldn't argue the point. Especially since the humble dagger is worth 20 arrows, lol!
But it sits at the crossroads of game balance and "realism". We can accept a wizard firing minor spells all day long long before we're willing to pretend people just have infinite arrows. Or rations. Or torches. Or lamp oil.
I've toyed with a "Hollywood Ammo" rule, where, just like a typical action hero, you never run out of ammo unless it's dramatically appropriate to do so (insert Harry Callahan quote here), but ultimately, I just do things behind the scenes, and give my players slightly less treasure to cover living expenses, ammo, rations, and the like. Because, like me, and I think a lot of people, the instant I make a big deal out of it, they'll go overboard with a solution, nuking the problem from orbit, lol.
I didn't announce this, and they haven't even noticed my "Verisimilitude Tax".