The closest D&D has ever gotten to something like this was the point-buy system in Player Options: Skills & Powers.
And that is a big clue as to why talent trees are generally a bad default choice for organizing all such options--the system is either class-based or it is something else based. You can't finesse the issue with talent trees.
The thing that I dislike about talent trees is when they have prerequisites that you don't necessarily care about for your character. That happened to me in our most recent Saga campaign and it was disappointing to be required to spend my character resources just to get access to abilities that emulated the character that I wanted to play.
Yep. And no matter how you organize it, someone gets left out. The whole point of trees is organizing and presenting opportunity costs. If those opportunity costs are not consistent with your campaign, then some character is left out. That is, it is the same limits as classes with a lot more fiddliness.
****
Neither of the above to say that there is not a place for talent trees. I can see them working quite well in limited usage. But much like feats (and 3E prestige classes and 4E powers, for that matter), they work best when confined to exactly what they are good for--not broadened as some kind of generic mechanic for representing choices for every character. That's always the problem with the ivory tower aspect of these kinds of mechanics--someone has to say, "just use X for everything to structure it, and those little annoying exceptions won't matter."
I have a counter suggestion. How about determine what actual abilities are available to a given class, or group of classes, and why? Then if the best way to organize some of those are talent tree, and another set is really feats, and yet another set is class abilities--use what works best and don't worry about it being 100% consistent across every class?
When you use talent trees, use them consistently. When you use feats, ditto. Heck, only using a given structure when it fits the subject matter well will help them be more consistent. You won't need as many of those little annoying exceptions.
