Yeah, and thematically it just doesn't make any sense which is where we can best make our decision.
A thief rogue being able to cast Vicious Mockery faster because their hands are fast is nonsensical.
The thing that costs time is the casting of the spell not the manipulation of the item.