I think "canon" is a silly concept that quickly breaks down for almost anything, and comes from an odd confluence of religious and legalistic thinking that weirdly was appropriated to discussing fictional continuities. In its application to fictional continuities it tends to involve wielding an appeal to authorial intention in a battle between the preferences of rightsholders to "legitimize" further derivitive works and what story aspects and entries in a franchise fans prefer. The appeals to authorial intention are rarely grounded in a consistent view of it, and generally more rhetorical moves than anything else. This last point is not to accuse OP of engaging in sophistry, but rather of trying to engage honestly in a discourse defined by sophistries. It is a hopeless endeavor.
Setting aside "canon" for a moment though, I agree that at the end of the day the Lord of the Rings really only has consistent continuity with itself and to a lesser degree the Hobbit and that these are the only Tolkien works which we have a published version approved by an author who was very particular about every detail and who vacillated wildly in his decisions on those details. As I'm very fond of mentioning, Sauron started his literary existence as a talking cat. While I'm confident we don't have the "talking cat" version of the Silmarillion vs. what an immortal Tolkien would have ultimately produced given an infinite amount of time, it is not a finished work which can be considered his definitive vision. And in the particular case of epic fantasy fiction on the Tolkien model there is particular reason to care about authorial vision, because there is just such a vast panoply of moving parts that only the author might possibly understand the full significance of.