I think it depends on what type of game your going to be playing. If your in a strictly kill the monsters, find the treasure type game, the yes, you wasted the points.
If the game is one where story elements are as important as game mechanics, then probaly not.
Another thing to keep in mind when designing a wizard is that it takes years of study to become a wizard, as opposed to sorcerers, who come upon magic without training.
Your character can still be knowledgeble about farming without spending the skill points.
From the PHB 3.5 pg62, character skills sidebar:
.....the skills on your character sheet don't actualy define everything your character can do. Your character may have a solid familiarity with many skills, without having the actual training that grants skill ranks.
An even better idea is to use the skill points you spent on farming on other skills that can be explained as having been learned while growing up on a farm. Take handle animals, knowledge (nature), ride, appraise, survival. All skills that are not on the wizard list, but are useable in just about every type of game.