An N95, at least a properly fit-tested one which actually seals, does indeed protect the wearer. But almost no one is wearing a fit-tested N95 in public. They're mostly wearing cloth or surgical masks, or occasionally K-N95s or even full N95s which just haven't been fit-tested and so don't have a proper seal.
Non-fit-tested masks DO provide at least a little protection for the wearer, but it's pretty minor. The degree of protection varies depending on the thickness and material of the mask, and how well-fitted it is. Which is also true of its ability to catch your own respiratory droplets to protect others, but that efficacy is substantially greater than the degree to which it protects you. Both are aided by distancing, which reduces the amount of viral load either party is exposed to, though if you're in an enclosed area with an infected person for an extended period (say, an hour+), distancing isn't much help.
So you're both basically right- masks are primarily to reduce our own viral dispersal if we are pre-symptomatic or asymptomatic carriers, with only a secondary role of offering us some minor protection against viral spread from others. This is somewhat distinct from how the fit-tested masks and eye protection worn by frontline healthcare providers are indeed primarily for their own safety.