How does no one notice? Nothing in the description says the mage armor effect is invisible. Sure, you might say nothing in the description says it "isn't" invisible either. But compared with Shield, another 1st level abjuration spell that provides a barrier of magic force to protect you, Shield explicitly says the barrier is invisible. See text below:
Mage Armor: "...a protective magical force..."
PHB 275 Shield: "An invisible barrier of magical force appears and protects you."
I don't think Mage Armor has to be neon-lights flashy and attract foes from miles away, but magical effects seem by default to give off perceivable effects suitable to their nature unless they explicitly say otherwise.