Except...
In, say, a flashlight that's turned on, you have a complete circuit, and current flows.
When you turn it off, the switch breaks the circuit, and the current stops flowing... much. There is still a potential difference (a voltage) across the switch. The resistance across that gap is sky-high, so very little current can flow, but very little is not equal to zero.
that's actually a good explanation for why batteries left in a flashlight that is left off also go bad after a long time. electrons be flowing in a teeny tiny trickle. Basically, switches are leaky spigots.