To clarify, despite the titular character "Xanathar" being from the Forgotten Realms, the book is in no way a Forgotten Realms book. I really want to emphasize this. There are books that are Forgotten Realms books, and those that are not. XGtE is one that is not. It's a general 5e game expansion, sort of an extra half of a PHB and an extra half of a DMG combined.
Sword Coast Adventurer's Guide is a Forgotten Realms book. However, it also has a plethora of information that is applicable to the general D&D multiverse that also includes Greyhawk and other previously published campaign settings. For instance, the non-human deities aren't Forgotten Realms exclusive. They are worshipped also in the Greyhawk setting, as well as hundreds of worlds in Spelljammer and Planescape. That being said, at this point, about the only reason to get it other than getting Forgotten Realms content would be to get the few exclusive class and spell options. Some of those are worth getting. In particular it includes the green flame blade and booming blade cantrips that they haven't reprinted anywhere, but I consider essential options warrior-mage types.
Along the same lines of Xanathar's Guide not being Forgotten Realms, Mordenkainan's Tome of Foes isn't Greyhawk, despite Mordenkainan being a Greyhawk character. It's a book that combines lots of information on the non-human typical PC races and their pantheons (re-presenting the material on those pantheons from SCAG, but with a different take on some of it), with some information on more exotic or monstrous planar creatures, and essentially the third monster manual for the game making up the second half.
The second monster manual is Volo's Guide to Monsters (another one that is setting inclusive despite having a named character, this time a Forgotten Realms one), which follows a similar setup to Mordenkainan's Tome of Foes, but with orcs, goblinoids and some classic monsters as the main types of creatures focused on in the first half of the book, and a chapter with a variety of stats for additional PC races (Mordenkainan's Tome of Foes also present a few new PC race options).
Interestingly, Ghosts of Saltmarsh actually is a Greyhawk book, although its touch is very light on anything outside of the adventures. And much like SCAG has some general character options at the end that aren't necessarily Forgotten Realms specific, GoS has a big set of ship and sea-faring rules that aren't specific to any setting.
D&D lore is a huge sprawling thing like comic book and sci-fi universes, so there's a lot to know. Just ask around if you aren't sure what is actually limited to a setting versus general D&D stuff.