I am thinking if the effect is not visible, the component hand waving and chanting just looks like mumbo-jumbo to a layperson who knows nothing about magic. Also, think about a spell cast while concealed or simply far away.
I think that this is about as likely as a zombie apocalypse breaking out in real life and people not immediately recognizing it as a zombie apocalypse.
In a setting where magic is real - or even just believed to be real - any hand waving or chanting looks like magic to the observer, even if it is meaningless mumbo-jumbo. And perforce, anyone that goes around muttering to themselves or otherwise saying things in public people don't understand is very likely to be thought of as a hostile spell-caster, and lynched on the spot. In the real world, where hardly anyone believes in magic, when a door closes or opens with no apparent cause, the first thing we think is that we didn't close it well and "the wind must have blown it open". In a world with magic, the first thing everyone will think is, "There is something invisible that has just entered the room with me."
Back to the topic...
D&D spellcasting is not meant to be subtle by default. This is one of the most difficult things to convey to a new player. The components of spellcasting are overt, obvious, and basically impossible to conceal without special training.
In a society that has to deal with people being able to cast curses, being able to use mind control, and otherwise being able to do horrible things to them with forces that they can't comprehend, they will not ever think that someone is just being goofy. They certainly don't in the real world where magic is believed in, and that will go 10 or 20 fold in a society that really has to deal with it.
In my campaign, it's the social norm for anyone that is going to use a spell to announce that he is going to do so, and receive permission from all those present. If anyone starts dancing around, waving fingers, and muttering in strange tongues without having made clear that something like that is about to happen, he can generally be expect to be met by lethal force from anyone present. No one will think the slightest of picking up a carving knife and trying to stab you, or kicking you full force in a sensitive region, and you shall count yourself lucky if you aren't beat senseless and wake up to find your eyes gouged out, your fingers broken, and your tongue removed. People in the setting consider "Charm Person" mind rape, it's punishable by burning, and people with a grudge will bring charges against a vulnerable spell-caster of trying to charm them - even if they haven't. So if you want to cast "Charm Person", don't do it in public and expect that if the victim passes their saving throw, they'll treat it as a vicious assault on their person and not some harmless prank.
A spell-caster can be subtle if:
They rely on Silent spells or spells which otherwise lack verbal components, or they use Disguise Spell to cast a spell as an opposed skill check with their sleight of hand versus the audience's spot (or perform versus spot in the case of a Bard). Silent-Still spells are even better, but its a lot easier to get into a dark corner and hide your finger waving, than it is to cast a verbal spell without a loud clear annunciation (without training at least).
They rely on long duration spells or spells which they can get a long term benefit from (such as divinations) which they can cast in private.
They rely on long range spells that they can cast from a distance while unobserved.
They rely on spells that disguise them in some mundane way such as shapechanging into animals.
However, D&D magic is as I said not really meant to be subtle or pervasive. It's designed and balanced according to its perceived utility in a dungeon/adventuring environment with a nearby Haven, and where most challenges involve tricks, traps, and combat with monsters. So it's notably not like the magic that people in the real world attributed to spellcasters when they believed in magic, which tended to be very subtle and pervasive.
Could you adopt D&D magic to make it subtle? Sure. You could remove all the splashy stuff and leave only the sort of things that people could hide, and you could make it cheaper to acquire feats that removed components from spells or use things ideas Subtle Spell and Disguise Spell to allow casting in public places with a chance of being unobserved.
You could also introduce new classes of spells with very weak effects but very wide areas of effect or very long durations - Hallow is an example from the SRD that sort of falls into this class, though its a bit on the powerful side. The problem that this tends to have though is that you end up with a world filled with fiddly hard to track buffs and debuffs.
Could you have a setting with D&D magic where people didn't believe in magic? Maybe. You could do something like magic has very recently been discovered, and no one believes any of it is real since for centuries everyone has been able to prove that magicians are just charlatans. Or you could have a setting where magic is say 1/1000th as prevalent as it typically has been in published D&D, and supernatural things are so vanishingly rare that most people will never run into them in their life time. The problem you are going to run into there is that D&D puts so much potential magical prowess into the hands of the PC's, that in a setting where magic is rare, they are typically able to dominate their surroundings and the NPCs. The monsters aren't tough enough, and the NPCs are too naïve regarding magic and have no defenses against it. It's a world where many castles are useless because no one made them with magic in mind, merchants are unable to realize when they are being scammed or defend against it, and people very likely won't recognize a spell is being cast when they see it, or even if they do they'll not expect anything to actually happen.
