The big question I want to ask is: what do your players think about this? Are they having fun with 5E as-is? If so, you have a problem. The DM gets to have fun like everyone else, but if the players want something you don't want to give them, that's a problem.
My suggestion is to take a look at...