Honestly I hadn't noticed that it only has a verbal component that does change things a bit and will require a rethink, however I never said anything about certain materials stopping it nor do I think would manacles (provided they aren't the full hand encompassing type) prevent a somatic portion of a spell being used. Nowhere that I can find does it say spells require your hands to be in front of you and imho if it did you would have a lot of trouble casting a lot of the old spells (and without knowing the full current spell list) and a lot of the new ones that deal with giving bonuses to NPC interaction "Your a friend right now but I just saw you wiggle your fingers at me and I've never seen you before in my life". IIRC in old editions the person became best friends.
At the end of the day how people see spellcasters, magic, and spells in the world changes what people think they can do. I had a conversation with a friend a few years back that really put it in perspective for me. Long story short he believed characters are born heroes while I believe characters become heroes. Magic could very well be put in that same light.
As an aside your pointing out that the spells doesn't say anything about manacles stopping teleportation it also doesn't say anything about being aloud to choose what you bring or that it doesn't bring certain things. It's two line descriptor really leaves it completely open to interpretation and assuming that it does what you want because of something unrelated (vocal component only) doesn't really support that.
I am fully ok with it leaving behind all of your clothes as well as the manacles, maybe even allow my players to pick. Keep everything, or keep nothing. My point before was simply that the Manacles being attached to a wall should have no effect on your ability to teleport, because it is teleporting and not moving really fast.
As far as Somatic components, I generally consider them a less flashy version of the Doctor Strange style casting, maybe more symmetric wrist movements instead of full arm movements. I am actually working on a "Dual wielding" style of casting, so a player can choose "Single Circle", as a power house/two handed style, which uses both hands in a single circle, or "Twin circle" which is weaker, but faster. This would allow things like casting two separate 1st level spells in the same turn. It is a very young idea, but I like adding more depth of choice to classes. All of that aside, a number of people I see on these forums considers Manacles a way to stop Somatic components, and it might even be RAW (Not that it is important outside of helping people understand RAW).