Agreed. Especially because the whole turn-by-turn, action economy thing is itself already an artificial imposition grounded predominantly in metagame considerations. The gameworld itself is not a stop-motion world.I have no problem with the idea that my character is calling on inner reserves in-world when he spends an AP.
With my group, one feature of APs that contributes to the ingame interpretation that you describe is that the players deploy them when things are getting a little desparate or urgent. So the players' emotions, at the table, matches the imputed emotion/experience of the PC. Which, at least as I understand and experience it, supports rather than impedes immersion.