I've seen the College of Glamor and College of Swords in play, and both worked really well. College of Swords is less of a team player than most, being more focused on personal combat. Forge Domain is fun for a Dwarf Cleric, but don't know if it's really powerful or not compared to other Domains. Oath of Conquest is solid (we've had 2), but can cause some inter-party conflict if the group tries to be heroic. Divine Soul is amazing, but tricky to run, since you have so few spells known. The Hexblade is awesome, replacing the blade option from teh PHB, and another player has enjoyed the Celestial Pact. I'm currently playing a War Mage, and while it's probably not as powerful as a lot of other schools, it's really fun.
All in all, if I had to pick one non-core book, as a player, Xanthar's would totally be it.