After that the players would try to talk the druid player into missing the game so I could run the character and the party would have the "smart" druid back.
That is too funny. There is a player in my group who is so stupid I WISH the DM would just run his character as an NPC. Perhaps I should try the same tactic your players did ...