I might be misinterpreting what you're getting at but I think you can avoid my issue by removing prerequisite talents. An alternative would be to keep prerequisites but make them more general. Instead of the Point Blank Shot Talent being a prerequisite for the Precise Shot Talent, have the prerequisite for the Precise Shot Talent be any archery talent.
Sure. But if you do that, then we aren't talking talent
trees anymore, are we? That's just a list of talents.
Really, my point is that the game uses classes as packages of abilities, or it doesn't. Classes enforce niche protection by having opportunity costs. If you can have 4 levels right now, and you pick 4 levels of wizard, you can't have any levels of fighter, and the stuff that goes with them.
If a game does that, then alternative structures also have prerequisities and enforce opportunity costs, or they don't. If they do, then they are replicating what classes already do. If not, then they aren't talent trees.
Or a game could decide not to use classes, as S'mon discussed earlier, using talent trees or something like them instead. That's a valid design approach, but not one that I think will fly in a D&D game.
Basically, I think Talent Trees are (over) sold under the premise that they can replace classes, which I don't believe can happen in D&D, or sold in that they can solve the problems of classes, when I think that stacked on top of classes they provide more trouble than help.
A list of "talents" as an alternate way to handle some feat customization is another thing entirely, one that doesn't suffer under those restrictions, and thus might be useful.