Depends on the campaign for me. I tend to be a little more lenient in FR or other more "high fantasy" settings, but in my own, Rokugan, or even Iron Kingdoms there are definite repercusions to casting in public. Usually it's awe mixed with just a little bit of fear for wizards, much the same if its a priest, depending on the nature of the magic, and the god (explanation coming), and for sorcerors it can end up getting nasty.
I have a feat called Subtle Casting in my games, which allows the caster to make a concentration check 10+spell level to cast under the breath, with subtle movements and little visual effect. This can be done on the fly, to prepare a spell that way is +1 level. No check required for that.
Magic is subtly colored by the source of that magic. Divine magic in particular is affected by this. A god of good and bunnies' magic would be full of sunmotes and warm fresh scents, whereas a god of evil and vampires would be reminiscent of blood and tomb smells. This is a subtle thing, not enough to require massive game mechanics, but the idea is instead that casual onlookers can feel evil or good magic, and those with spellcraft might be able to know its source. For sorcerors their main concern is whether someone realizes that they did not learn to cast magic from a reliable source. Sorcerors have had a history of ending up on flaming sticks in the past.