Yes, I tend to agree. House rules that are contrary to the basic spirit of the game are probably a bad idea.
A possible alternative would be to make the first attempt at casting have advantage, second attempt normal, all subsequent attempts at disadvantage. That keeps the unpredictable element...