A shield is attached via depending on the model, a forearm strap and a handle or secondary strap of some sort, or just the 'handle'. It is most definitely in your hand though, otherwise you would have very limited control over it (even as a purely defensive item).
Also, a shield can bash you coming straight on at you, thus in no way negating it's ability to be in the way and thus help your ac. A shield bash isn't necessarily a swinging bash but can easily be a frontal shove.
Devil's advocating here:
Large shields are often as much strapped to the arm as held. Page 146 of the PHB has putting on or taking off a shield as an action, far more than unsheathing a weapon. It's not as simple as grabbing it with your hand.
And by the logic of "it's on your hand" I should be able to make an improvised weapon attack with a gauntlet, always doing more damage than a regular unarmed attack.
And there's still the question of whether it would still grant AC. The shield's in the complete wrong position to defend you, being pushed away from your body. And your weight would be set for pushing with the shield and not positioned to receive blows.
At the point where the guy who drafted the rules you're citing disagrees with you, I am gonna go with the guy who drafted those rules.
Crawford's special authority over the rules ended when it went on sale. His opinion currently benefit from his having a better understanding on the background of the rules - having had a hand in their development and seen how things work - but that advantage is quickly ending as more people play the game and discuss the game and break apart the game.
Soon, giving Jeremy Crawford's personal opinion weight would be like deferring to the DMing advice of, oh, say, James Wyatt. Mr. Wyatt worked on D&D but is now on the MtG team and might not necessarily have greater knowledge of a rule than you or I.
Quick responses to a question fired out on twitter are not an official FAQ, nor even actual Sage Advice content. And even if they were on the website, the Sage Advice articles make it pretty clear they do not overrule a DM.