I can see the scenario now using these "simple" rules.
DM: "OK, as I explained I want to have a grittier campaign so we're going to try out some armor damage rules. Please review the sheets I gave you. Last we left off you were investigating odd occurrences at the circus, you suspect clowns are handing out cotton candy causing madness."
Pat the Paladin: "I hate clowns!"
Barry the Barbarian: "Huh. Guess I don't care about these rules. Awesome to be able to deflect arrows with my pecs!" Throws the piece of paper with rules away.
Clary the Cleric: "So we have to keep track of armor damage points? The armor just falls off one day? Not sure that's realistic."
DM: "They're preliminary rules, we'll get to that later. Alright, you were headed to the cotton candy station run by the clowns when hear an ungodly trumpeting sound and people screaming in terror. Roll for initiative."
DM explains that young children, faces covered in cotton candy have been throwing handfuls of peanuts at an elephant which is now rampaging. The children see the adventurers, start throwing peanuts at them while the elephant charges*.
DM: "OK, Pat - you are pelted with peanuts! Does a 15 hit?"
Pat: "Nope, you need a 20. I just got that new armor."
DM: "Pat, you take 1 point of armor damage. Clary, does a 19 hit?"
Clary: "Yep, I have an 18 AC."
DM: "OK, you take 1 point of damage. Barry, does a 13 hit?"
Barry: "Nope, bounces off. Because I'm tough. Oh, and I don't take any armor damage because I'm not wearing armor"
DM: "Alright, the elephant charges Clary" [dice are rolled, she's gored and stomped on for a total of 41 damage]
Clary: "Cough, cough ... OWWW! That hurt, how much damage did my armor take?".
DM: "No armor damage."
Pat: "Wait, she gets gored and stomped on by a frickin' elephant and her armor takes no damage but mine got hurt by a handful of peanuts?"
Barry: "Dude, told you barbarians are superior."
To me that does not sound "gritty" or "realistic". If you can't accept the gaping holes in your proposed solution there's not much anyone can say. Throw out some high level proposals and we can help flesh them out. Maybe something along the line of taking damage to armor after taking
n points while tying it HP and type of armor.
I'm not totally opposed to armor damage, but the current concept just doesn't work no matter how much you tweak the implementation.
*Example based on a home game scenario, names have been changed to protect the innocent.