I've found it easy to buy the right food without busting the bank (unless you are trying to eat organic or something).
Interestingly, in my experience (and I can only speak for myself), I found the whole "organic is too expensive" thing a fallacy.
I started changing my diet, gradually over a couple of months about a year ago. I was pre-diabetic, grossly obese (though I would never admit it before), my blood pressure was 165/110, I had a fatbelly, fatface, multi-chin, my cholesterol was horrible and my triglycerides were like 700. I was a dead-man walking, and only 8 years earlier I was in the best shape of my life, even better than when I wrestled in high-school. I found that I leaned on things, I would get winded going for walks with my wife and daughter, and I couldn't even comfortably reach around and wipe my own damn ass (sad, I know).
I had to have a moment of clarity. I saw myself dead before I could play with my daughter, I hated going to the pool or beach so I would miss these opportunities. I had always dreamed of coaching my daughters softball/swim/whatever team and there's no way I could have participated in that. I loathed what I had done to myself.
So first step. Change my diet. It was something I could do, and I coupled it initially with progressively longer walks.
I found that when I bought organic free range eggs, organic or grass-fed beef and chicken, wild caught fish, fresh organic fruits and veggies (though maybe a bit less so with fruit and veggies if I buy from the local farmers market to support local businesses), organic cottage cheese, butter, milk, yogurt and other dairy, blah, blah, blah..... And eating these things in appropriate portions at the right time of day, I realized that I stopped spending money on chips, sodas, boxed foods, cookies, cakes, sugary popsicles. etc.
After doing a grocery bill analysis after 6-months (thank god for personal finance software that lets you categorize expenditures), I found that I spent between $50-$100 less each month! And the refrigerator was always full!
I didn't totally do away with tasty treats either, there are a LOT of yummy things you can make with fruit, yogurt, organic skim milk and an ice cream maker, as well as a few popsicle molds. And you can get some organic cookies that aren't unhealthy.
My life is dramatically different after a year (60 lbs lighter), and I have begun a recent exercise program that I hope will take me that last 30-35 lbs to my old fit self. I have a naturally muscular frame, though I am short, and I can retain muscle mass well, but I always had difficulty with lean abs look. So I now have a Nov-Dec goal to get there.
At my last doctor's appt two weeks ago, my cholesterol is under 200 with my triglycerides under control, my blood pressure is 130/75, my fasting blood sugar is 75 and I am officially no longer prediabetic. I have work to do still, and has been including more rigorous exercise, but my wife finds me sexier (sex life is 1000x better), I can run around with my 3yr old daughter all weekend, my back pain has gone away, and I sleep deeper and more consistently at night. I stopped popping antacids like candy, taking prilosec every night and I have been taken off my cholesterol and blood pressure medication.
(Bullgrit, I am working up to P90x, I am halfway through their easier Power90, though for a guy like me it still kicks my butt every day and I am already doing the advanced group of exercises - Phase 3-4. P90x will be started sometime this summer.)
I know that was a bit of a tangent, but I wanted to point out that the costs of NOT going organic or healthy are way higher if you are out of shape like I was or even just chubby. Don't let the cost deter you, you will SAVE money over the long term.