The counter-reading is:
Scrolls don't require you to take the Magic action. They require you to cast a spell.
Using the Magic action isn't a requirement of the item. It's a requirement of the spell you're casting (evidence for this is that bonus action spells and reaction spells don't take the Magic action, because they take your Bonus Action or Reaction, which isn't the Magic action).
It's not a requirement of that category of magic items as a generalization. Much like you can have a wand which casts spells as a bonus action, or a magic action, or even a 1 minute casting time in theory (though I don't recall such a wand).
However, it's a requirement of that specific magic item, if that specific item requires the magic action to trigger it.
Examples:
Wand of Radiance: "A creature that is attuned to
Radiance can use a bonus action...to cast the enhance ability spell, choosing itself..." As this specific wand does not require a magic action to activate it, it cannot work with fast hands.
Wand of Magic Missile: "you can use an action [now called a Magic action] to expend 1 or more of its charges to cast the magic missile spell from it. For 1 charge, you cast the 1st-level version of the spell. You can increase the spell slot level by one for each additional charge you expend." As this specific wand does require the magic action to activate it, it can work with fast hands to reduce the casting time to a bonus action.
So while the category of magic items "Scrolls" does not require that you take the Magic action, a specific scroll does require you to take the Magic action. The distinction of "to cast" does not seem relevant as all magic items which we all agree are supposed to be covered by Fast Hands, like a Wand of Magic Missile, also uses the term "to cast [the] spell." Most magic items which could be used with Fast Hands cast a spell, and the magic item user is designated as the caster most of the time.
Fast Hands only applies when the Magic action is a requirement of the item.
A spell scroll of magic missile, like a wand of magic missile, both require the Magic action by that specific item. Both "cast the spell." It's not tied to a category of magic items, which is just a title used to organize magic items. It's tied the requirements of that specific item.