Having been in more than my share of street fights and done a little amateur boxing, nope. Evading attacks is far easier and far less taxing physically than actually throwing punches, grappling & kicking.
Actually, wearing armor affects your overall speed and coordination, so yeah, if "realism"...