I'll add my vote: yes, it's a waste of skill points.
Two tenets of D&D are involved here: 1) skill points are more precious than gold, 2) D&D PCs work best when they specialize.
Ergo, skill points need to be spent on skills relevant to your class, as well as spent on maxing out key skills your PC chooses to focus on. Spreading ranks across too many skills, or spending a lot of ranks on a skill that's there purely for flavor, is a waste, and it makes your PC less effective at what they do.
Roleplaying flavor is fine, but you don't need to waste skill points for roleplaying flavor. If you want to have your PC's background as a farmer matter, all that PC needs to do (mechanically) is be able to beat a DC10 (average) P:Farming check. Since I can probably guarantee you that any situation involving a P:farming check will be one in which you can Take 10, at most you need *1* rank or, even better, to have your DM allow you to use the skill untrained. Unless you have a negative Wis mod, you'll be able to handle average farming-related tasks, no skill points wasted.
The idea of getting other useful skills (e.g., Handle Animal, Knowledge(nature), etc.) as part of your background is also not a bad idea.
I tend to be vehement about this because one of my current groups has at least half the players spending a lot skill points on "flavor" skills. E.g., our dwarf cleric has ranks in Craft (stonecraft) and Craft(stonecarving)... even though there's no reason, in D&D, to differentiate between the two. It drives me nuts because our DM is running a pretty combat-heavy, rp-light campaign, and having key spellcasters with *no* ranks in Concentration or Spellcraft because they spent all their ranks on flavor skills and Craft, which have yet to come up in about six momths of gaming, is pretty frustrating.
This is all reminiscient of the thread on disadvantages in d20. If you want your PC to have a background as a farmer, write "Background: farmer" on their character sheet. Any DM worth their salt will let you make use of it without you blowing skill ranks on useless skills.