I can't speak for Treebore, but I would have preferred to see a game that still had the core gameplay present in OD&D through 3rd Edition. That would mean fighters who still play like fighters; wizards who still play like wizards; and so forth.
All of these things remain true of 4th Edition.
This is bordering on threadcrap at this point, but what I state is an objective fact of reality.
To boil it down to a basic and easy-to-understand element: Pre-4E fighters featured virtually no daily resource management. Pre-4E wizards, at the other extreme, featured gameplay that was 99% about daily resource management.
Neither remains true in 4E: The gameplay for both fighters and wizards feature a mix of daily, encounter, and at-will powers.
Using something similar to your X/Y/Z example:
A - Players that would play 3rd, but not 4th (or prefer 3rd)
B - Players that would play either
C - Players that would play 4th, but not 3rd (or perfer 4th)
At the moment, they are 'giving up' on A, and splitting the B group. If they were to make a new edition that was very similar to 3rd, they would completely ignore the potential to bring in C, and fight with the previous edition over the combined A+B group.
Your analysis is flawed because it ignores the other option: The hypothetical 4th Edition that remained true to the core gameplay of OD&D through 3rd Edition. Let's call it Alter-4th. That gives the full list of categories as:
A - Players that would play 3rd, but not 4th or alter-4th
B - Players that would play 3rd or alter-4th, but not 4th
C - Players that would play 3rd or 4th, but not alter-4th
D - Players that would play alter-4th, but not 3rd or 4th
E - Players that would play alter-4th or 4th, but not 3rd
F - Players that would play 4th, but not 3rd or alter-4th
G - Players that would play any of them
Or, to sum up: The sales of this hypothetical 4th Edition (faithful to the core gameplay of previous editions while fixing the problems of 3rd Edition) could potentially include (a) gamers returning who wanted a fix to the problems of 3rd Edition and (b) a higher conversion rate among existing 3rd Edition gamers.
I think it possible that the the number of gamers willing to try this hypothetical alter-4th Edition after rejecting 3rd Edition would be smaller than the number of gamers willing to try the actual 4th Edition after rejecting 3rd Edition (because the game is so radically different that it appeals to types of gamers who had no previous interest in D&D). But, OTOH, I think it equally likely that the number of gamers willing to convert from 3rd Edition to alter-4th is probably substantially higher (because, by not being radically different, it continues to appeal to gamers who were alienated by the changes the actual 4th Edition made to gameplay).
So the real question remains: Is the number of people you gained by choosing 4th Edition over alter-4th larger than the number of people you lost by choosing 4th Edition over alter-4th?
There's really no way to know.
None of us has ever seen a game printed on a book that is perfect from the outset, so quality ain't what sells a game system...
It's clearly not the only thing. But, OTOH, arguing that "nothing's perfect, therefore perceived value has no relevance in the marketplace" is an absurd claim.