Sure, but then it specifies the actions things are associated with. Weapons are part of the move or attack action. Magic items that require the Magic action are part of the Magic action. Items which specify they require the Utilize action are part of the Utilize action.
A magic item which requires the Magic action includes the free object interaction fort that magic item in the Magic action used with the item. Scroll, Wand, Staff, etc, these all take a Magic action, and the object interaction is part of that Magic action and not separate from it.
Now if you are going to hit someone over the head with your scroll, it is part of your Attack action. But if you're reading it to activate the magic item, it's the Magic action. The action associated with your use of the item is what the "free" action is associated with, but there is always some sort of non-free phase it's associated with.