Hell, you might as well be holding a Breastplate +1 in your hand and argue you should get the +1 bonus from the breastplate! Sorry, it just doesn't make sense to me and does break my suspension of disbelief. (To be clear, if you held the breastplate and took the Use an Object action to put it between yourself and an attacker--maybe, maybe I would allow the +1 for the magic armor... maybe.)
Yeah, no I still wouldn't follow this. If you had a shield +X slung on your back, you aren't going to get the +X to your AC. AC is too important in 5E due to BA, etc. so I wouldn't allow it.