That adds mental load and is unnecessary complicated. It is way easier to fix the Actor Feat, then to implement a general rule that forces you to reference all the specific rules.
Maybe I make that more clear with an example:
The general rule for movement trough another hostiles creatures space is, that it is not possible, unless the creature is two sizes larger than you.
The specific rule of the Halfing is, that the halfing can do that, even if the creature is only one size larger.
So, anybody on the table knows, I can't do X per general rule. Halfling says, my specific rule says, I can do X.
Everything is fine. Only yhe halfling player needs to know the nimble rule, the other players don't.
Now with the Actor Feat and the implementation of the house rule, it goes like that:
The general rule for ability checks does ot apply to do X.
In order to do X by the general rule, you need feat actor.
If you want to do X without the feat, there is a new rule, you do ability check with disadvantage.
So, now every player who doesn't have the Actor feat, need to keep in mind, that in order to mimic speech, they can have to.use this new special general rule.
That is way more complicated and it gets way more frustrating to do, if you not only have to do that for mimic speech, but for other things as well.