You shouldn't need to houserule. PHB 195, you have ranged disadvantage if within 5 feet of a hostile creature.
I meant, when shooting at a target and the target is in melee with an ally (not necessarily you).
I call your bluff.
If you really mean to persist propagating this myth, Flamestrike, at least be honest enough to admit "I dont want to rehash the whole 'the game is balanced around a six encounter adventuring day' argument again, but thats how you 'fix' it: with lots and lots of DM prep time.
Its not my fault you stat up encounters and leave it there. Spend an extra 5 minutes to design an actual quest for the party (a reason to engage with your encounters) replete with some form of time or pressure limiting factor that pushes the players into longer AD's.
Its not that it cant be done with a minute or twos thought when doing your mid week adventure prep as DM. Its that you cant be bothered.
If you want to just stat up a dozen encounters, and scatter them on a map with no thought as to why the adventurers are there, what they are trying to acomplish, and what time or other pressures are working against them, thats not my fault, nor is it a fault of the game as written.
What advice do you give a DM that purchases an official module to not have to create her own adventures?
Both the entire stories and around half the actual encounters in the published adventures to date include either pressure constraints [youre being pursued through the underdark, the caste is under seige, youre invading an enemies castle etc], time constraints [break the seige by midnight, stop the demon lords/ elemental princes before they power up] or both.
You simply refuse to police it. If your players are relying on the 5 minute AD, have the demon lords/ drow house/ elemental prophets/ Strahd come looking for them (as they certainly would). It gives the PCs enemies the time to locate, engage and destroy the PCs, relocate and reinforce or whatever.
Imagine youre in Afghanistan as part of a 5 man special forces team, behind enemy lines. You honestly think youve got all the time in the world to complete your missions, being able to kill the guards of a stronghold and then fall back to camp for the night? Or sneak into an en enemy camp and capture an enemy leader with no repurcussions, time constraints (how long he will be where your intell tells you he'll be there) or any other factor involved? Or do anything at all that doesnt involve a time or pressure constraint?
Every mission you ever take on in the Army has a time limit, and key timings. Be at place 'X' by time 'Y' to do task 'Z' which needs to be acomplished by time 'XY'. Just like every other job in the world. If you dont complete the contracted work by X you dont get penalised. If you miss deadline X then bad thing Y happens. If you hit targets X then you get bonus Y. Etc.
All you need to do as DM is use this in your games. Rather than break verisimilitude, it enhances it. It makes the world seem both alive and real, and it drives the story and provides dramatic tension (in additon to its balancing game effects).
I have no idea why you find this so hard to police, or why it makes you so angry doing your job as DM.