A shield takes an action to don or doff, so I think it is worn.
OK, so what are we discussing here? If we're talking about the "official" rules, that's already be clarified explicitly in Sage Advice. If we're talking about your house rules, then I say go for it, whatever works best at your table is what you should do.
The interesting question to me is, based on the regular game rules, how do you decide if a particular item could be used as an improvised weapon, or whether it would be incorporated into an unarmed strike. You seem to suggest it would be based on the effort needed to equip the item. That doesn't seem very compelling to me, nothing in the rules language sounds to me like it touches on that.
I'd say that a more straightforward interpretation would be whether the item requires your hand. For instance, if you played that a shield was strapped to your arm and that your shield hand was free to hold another item, then I'd be inclined to agree that attacking with your shield would be an unarmed strike. But the game rules say you need to hold the shield with your hand, regardless of any other straps or whatever. On that basis I think you should count the shield as a held object useable as an improvised weapon.
To return to your previous questions: if you were holding your helmet in your hand for some reason, I would certainly let you make an improvised weapon attack with it. If you were holding your gauntlet in your hand, I would probably say it was too small to use effectively that way.
I prefer that interpretation because it lines up better with the notion of holding, and is also mechanically consistent... unarmed strikes don't require a hand, but do less damage.
If you rule that it is held, do you then allow disarm attempts on a shield?
Well I don't play with disarms anyway, but if I did then yes... but. My picture of a shield is that you have to hold it with your hand to be effective, but it is also strapped to your arm. So I would say that if you disarmed my shield, you would knock it out of my hand, but it would still be hanging on my arm. I would not benefit from it's AC bonus, but you couldn't just pick it up either.
So if you like, you might say that a shield in particular is both held and worn. I think that for such an item, the holding part means that it can serve as an improvised weapon (assuming it has a suitable size and construction).