Do it! Then the adventurers need to come across a farm run entirely by animals, where the sheep bleat "Four legs good, two legs bad", and all animals are equal, except the pigs are more equal than others.
In that game, the characters are animals, so some other considerations are more significant (but have not been unbalancing); namely, simplifying/eliminating material components (somatic components are fine--ever watch a dog circle three times before going to sleep?) and reworking the stat ranges for animals to something playable.
The ability to talk has never been a problem, whatsoever.