It's a social game above everything else. If your table isn't having fun, then find out why, because that's the most important thing. Not the rules, not the dice, not the game. Having fun with friends is. And feel free to change anything you want if it means making the game more fun. You're the DM, which means you have a lot of control. That also means take the time to get prepared. You don't have to memorize every rule, but you should have a good understanding about how the rules work, and how your NPCs/Monsters would act in the game along with their capabilities. Don't allow yourself to be bullied by rules lawyers or players who think their fun is more important than anyone else's. The best way to stop these sorts of players is to ask them, "Your ideas sound interesting, would you like to DM instead?"