Yes, they store spent fuel on site. For one thing, it is loaded with stuff that has half-lives of ten thousand years and more - a decade or two here or there doesn't touch the long-term problem. Plus, while the site is in use, it is perfect for securing radioactive materials. And, after it ceases producing power, it will take about 60 years to fully decommission the site. You have a long time before that spent fuel has to find another home.
And, perfectly honestly, if we wanted to, we could recycle the fuel. We don't for legal reasons that have nothing to do with power generation. We could change the law, recycle fuel and the end product of that chain is much less problematic than what we have now.
What you're really glossing over in focusing on the fuel is the real problem of disposing of the
reactor building. It has been there, being bombarded with stray neutrons and alpha particles for decades as the generator operates - the building itself becomes radioactive, and has to be taken apart and disposed of.
While by some measures each pound of it isn't "very" radioactive, what you lose in intensity you make up for with there being thousands of tons of the stuff. Modern nuclear reactors have something like 40 metric tons of steel and 190 cubic meters of concrete per megawatt of average capacity. And a typical reactor produces a gigawatt - a thousand megawatts. So, there's a lot of material to deal with. You can't just jackhammer it apart and pile it up without contaminating the countryside with radioactive dust - so, there's decades of decontamination and cleanup.
And... dump nuclear waste in the ocean? Because, what, the ocean is infinite and our power needs are and will forever remain minimal, so the ocean will never notice? We used to think that about the atmosphere, too. See how well that turned out?
Of course, solar/wind is (or is becoming) cheaper per Joule. It just can't provide power when people want to use it.
Oh, come on. There's loads of ways to store that power for when people want to use it, with very little loss of efficiency. For example - use the extra power generated in low demand times to pump water uphill into a reservoir. Use the reservoir to drive hydroelectric turbines when you need power later. Use gravity as your battery.
The thing that our reliance on fossil fuels has taught us is that we cannot rely on
single sources of power. You can't just use nuclear power. You can't just use wind. Or just use solar. If you go all-in on only one source of power, the issues and impacts of that source become unmanageable. When you break up power generation over as many different sources as possible, the impact of each is minimized.