Are components worth the complexity? Yes, for a variety of reasons that have been pointed out. My personal favorites: 1) you can bind and gag the caster prisoner and 2) you can't just "sneak off" that meteor swarm.
Are separate, specifics V, S, M components worth the complexity? Meh, occasionally. I like that the wizard still has a couple of tricks if he's only bound, but not gagged. Over all, though? Not really. In 95%+ cases, there's no functional difference between "they can hear you cast" and "they can see you cast". Honestly, other than the power word spells, you're probably better off binding and blind-folding the caster than binding and gagging him. Almost all destructive spells need line of sight, IIRC.
The biggest rub I have with VSM is that I finally realized that what I used psionics for in prior editions would be better represented by a sorcerer sub-class that used different trappings than the new mystic class from UA (the mystic is a fine class, just a bit different). Going back to my first paragraph, how does one keep the same checks on a psi-sorcerer without using VSM? Ditto for "spell-like" abilities.