Darn good question, actually. I have two children at the age where they are hearing more swearing at school. But, there is no way I'd let them read Martin right now, but that is as much for the sex than the swearing.
Here's how I view swearing in general: it's lazy and easy. There are times when it is appropriate for a character(s), and there are times it is not. Frankly, it isn't actually all that polite to swear. Just as I teach my kids to chew with their mouths closed, despite the fact that others are chewing with their mouth open, I teach them not to swear.
An occassional swear word in a book isn't going to kill a child, but constant use requires a parent to engage in a "teaching moment". Those aren't all bad, as discussions about Huck Finn demonstrate, but you don't need constant swearing. There are plenty of good books without it, and they aren't all censored, just as there are good books with it. I admit that at times I tire of it even in Martin's books, books I consider great reads.
As for those that say that others shouldn't be offended by some language, that isn't up to you. People are free to fee offended anytime they want. They may have religious, or culutural, or just personal reasons for feeling offended. An author or speaker needs to be aware of her audience, and to make choices about theme and words based upon that audience.
All you saying you don't mind the words, just curious, do any of you have kids?