Yes. And when it's successful, it'll likely wind up as a fun, memorable story.
It's actually never happened at my table. None of us have ever thought to try it. So I don't know how spellcasters would react. On top of that, I don't use very many spellcasters as a DM, either, so this would affect very few encounters.
If you describe how you might do so when taking the shot, then yes. I suggest telling me you aim for a buckle or strap.
Something i often try to keep in mind, uses of common capabilities are not "invented" the first time a player at my table happens to do them.
The first time one of your PCs used a magic missile to damage a foe was not the first time that happened in the verse. the first time one of you guys used invisibility to sneak behind a set of guards was not the first time it had been done.
Same for "disarm spell pouch" since that is just using a "assuming you allow it" standard combat maneuver against an obvious target.
Not talking "we invented new spell" or "we discovered new item" or anything like that.
So when it comes up you should, well IMO always should - i do, look at it in context to "what does this mean to my world as it stands and has existed" and then rule on how that works.
As for the armor and stuff, maybe you have quick release armor in your world, but in DND5e taking off armor willingly takes a while, so that is a fine setting element to add with the expected consequences. But one should keep in mind this approach also sets you up for "stranger danger" where less obvious "how does this work" kinds of stuff can tend to get rather oddly imbalanced results.
The pouch is described as a small pouch etc... does not list any sort of protections and this and that but has lots of in game impact. Well, just like armor was not described as "held on with one big vulnerable buckle" but we have ourselves a fairly strong idea of what armor was like to varying degrees so we have a foundation to rule on "how hard is it to get out of" or specifically "how hard is it to knock it off someone with an arrow."
i view practically every depiction in the rules as "in general" and "shorthand" especially for the arcane/divine/otherworldy types of stuff.
they dont talk about how many buildings or strongholds have wards and what they are in the DMG and PHB but i assume many have them in a magical world. i never assume the spells in the PHB or DMG are the sum total and that there are likely lots of other "deterrents" lying around to make certain things chancy... and a little research from the party can start to shown that.
heck a trip into a "bank" in a sizable town or city will likely find it in a temple or such and see the big glowing orb in the main room with swirling shadows in the glow. A sure sign that trying something is going to be a lot less dull than say jumping our IRL mobile home bank on wheels or ATM in the boondocks.
So, net result, as you determine what the rules and results are... if it were me i would be looking at it from "how the world has been dealing with it.
As i have said, might be worth considering if mage armor and other similar spells have an "protective cover" effect for arcanely related items - like spell pouches. Maybe even something special for attuned items as well. giving everybody a bit of the ole witchery switchery to protect a bit of their more significant character defining investments. it wasn't like Elric and Stormbringer (or Arthur and Excalibur) or most of the other heroic figures spent their combat scenes getting disarm shots aimed at them by every tom, dick and ruffian with a bow, right?
For me, especially for attuned items and spell pouches and that kind of "critical to function" gear, i will not use the same "disarm" capabilities as i do for routine normal every day stuff cuz i don't see the combat and world we are trying to play in as what amounts to big games of "tag you're it" or "keepaway". Likely also a number of other mostly mundane options for solving the issues in other ways.