For me, I care not for historical accuracy. If I can put a jungle right in the middle of a frozen tundra and defy geography, why be concerned with history?
Same with tone. Guns can work even in a fantasy setting. And like it was said previously, I like things that go boom! For instance, there's no fantasy analog to the shotgun. The shotgun just has perfect sounds, power, and raw damage.
Ultimately, the problem comes down to balance.
1) Time. Others reference the black powder reload issues. It becomes almost pointless to reload when a combat could be over in 3 rounds if you fire the first round and spend the next 2-4 trying to reload. Not to mention the AoOs you're drawing.
2) Cost. This is another big limiting factor. Bullets or powders or the guns themselves cost gold per shot. Meaning that if you want to just compete with the archer in the party, you're going to be broke eventually just funding your attacks.
3) Damage. The logic applies to guns in general, but also to damage: A gun should be different than a bow, otherwise what's the point? Why go to all the effort to make, sink money and loading times, just to get an equivalent damage result? And if you're going to increase the damage, you should make the user suffer.
If these could be solved - make them slightly different without being a waste of tiem OR overpowered - would make me happy.
Then I would solve the in-game problem by making them rare. Perhaps their existence angers the Gods or the Spirits (in Exalted, guns frustrate the Spirits because they can't decide who's in charge of guns: fire, since it burns? Earth, since it makes the metal? Etc etc. So using them is taboo in a spiritual sense). Perhaps they are seen as barbaric, offensive to honor, or too dangerous to be allowed to exist. Perhaps, as someone else suggested, they are bygone relics - no one knows how to reproduce them, and so they are coveted but rare.
Maybe they are, inherently, magical. Just another type of magical device. So it's not powder and a ball, but a dart being spat amid lightning.
It doesn't really matter, as long as it allows the occasional PC to play a gunslinger, squint-eye staring at a group of orcs thirty feet away, a hand twitching to reach for his weapon.