Everything else! ^^
No, really. I can live with the fact that fictional settings have objectively observable supernatural abilities. But I expect such abilities to work in a way similar to natural abilities I am used to. Exhaustion is something I understand. If you cast too many spells, you need to replenish your energies, which may include taking a nap. I can also understand a short breather, that requires people to have a short time passing by before they can perform an exhausting action again. Mana points and cooldown are things that are similar to natural abilities I know from the real world, so they seem familiar and I understand how they are supposed to work.
But there is no mechanism in nature or technology of which I am aware, in which an ability can not be used even thoug the mechanism that creates it, and the fuel that powers it, are still present. If I have the ability to run, jump, and swim and after having made some jumps still have the energy to run or swim, then I also have the ability to jump some more.
I think the concept of vancian casting is that you meditate to form a kind of ball of mana that will cause an effect when it pops, and it is stored in your brain. Since space in your brain is limited, it can hold only so many balls of mana, and since forming these balls takes a long time you can't just do it on the fly. It makes sense. But does not have any analog in nature or technology.