In my limited experience putting this into practice, I think using either the flanking or facing rules in the DMG help with this issue not by penalizing ranged fighting, but providing incentives to go into melee. With facing, the rogue can get free advantage on any enemy whose back is turned to him. However, to avoid this, the enemy simply has to move around to the other side of the rogue's ally with whom he is currently engaging, and now in order to get advantage, the rogue has to circle all the way around the enemy; if he the rogue were in melee, instead, he could get around the enemy's backside far more reliably.