Fanaelialae
Legend
I disagree about the first part of your statement. The action is not being used to activate a magic potion, it's being used to drink the potion, just as you'd drink a vial of water. Actions to activate items are listed in the magic item description - for instance, an wand requires an action to activate the magic. I do not think they mean using an action to pour something out of it - that can be done normally with fast hands I think.
You're free to interpret the rules as you wish, of course, and normally I'd agree with you (it's logical that if you can fast drink an anti toxin you can do likewise with a potion). However, I honestly don't see much ambiguity on DMG pg 141. Under the heading of Activating an Item it says that you can't use Fast Hands to activate magic items. Then under the sub-heading of Consumables it says that "Some items are used up when they are activated. A potion or an elixir must be swallowed...". As I read it, RAW clearly states that drinking a potion constitutes activating a magic item.