Here is a house rule that I made up for my 3.x campaign. My group has been using this rule for ten years or so.
My House Rules said:
Firing Into Melee. There are two ways to fire into melee: carefully and carelessly. In either case, an attack roll of a natural 1 is subject to a Fumble Check. Also, in either case, if a creature provides cover to the target creature and you roll within the bonus provided by the covering creature, you roll again against its normal AC to see if you hit.
If you fire carefully, you are -4 to hit as per the PHB, and can only accidentally hit a creature if it provides cover to the creature you are targeting.
If you fire carelessly, you do not take the -4 penalty, but if you roll a natural 1 through 4, you have a chance to hit a random creature within the target area (even if the modified roll would have normally hit the creature). A natural roll of 1 is subject to both the Fumble Check and the random hit. The DM rolls to determine which creature adjacent to the target might get hit. The attacker immediately rolls another attack roll against that creature’s AC. Again, cover applies.
Note: It is possible that another 1 to 4 could be rolled against the new target, and then you would start the process of determining the random adjacent target again. If this happens, it is possible that you could get the shot back at the original target!
The Feat, Precise Shot, if used, negates the choice of firing carefully or carelessly.
The default is firing carelessly unless you announce otherwise.
Recovering Arrows. 25% of Magical Arrows that are fired and miss their target can be gathered (if possible) and reused. Any non-magical arrows that hit are no longer useable, as per the PHB. 50% of any missed, non-magical arrows can be recovered after combat (again, if possible).
I want to pre- defend the complexity of this house rule. I have found that, by the mid levels (7 on up), unless you are outdoors (or in a big cavern complex), the only characters that use missile weapons, are characters that take Point Blank Shot and Precise Shot anyway. Other instances of this rule coming into play usually involve a ranged touch attack with a spell, or where there are 30+ low-level opponents peppering the PC's with sling stones or arrow fire.
When it does come up, most characters fire carelessly, and then if they roll the 1 to 4, the DM determines the random target. The reroll rarely comes up bad again (20%), and when it does, the odd shot seems to engage the players as we roll lots of dice trying to figure out just where the arrow went. While this does slow play some, I find that in use, it is usually quicker than adjudicating some spell effects.
I have had more than one occassion where the bad/low roll happens twice, the arrow hits the original intended target, and then even ends up critting it with the rerolled attack. I think this is fine and for comparison, I have built into all of my Fumble charts: melee, ranged, and magic, a chance for the 1 to actually be a critical hit, but with some mishap to the roller.
The biggest challenge for the players is dealing with multiple covers. For example, if the target's AC is 18 (T below) and there are two sources of cover (C and C1) between the shooter (S) and his target, the closest cover to the shooter is hit on a 18-21 (C1). The next farther source of cover (C) is hit on a 22-25, while you need a 26 to hit the original target. Remember that the roll might only
potentially hit cover, and only if the roll overcomes the cover's AC.
Example positions:
T.......C........C1...........S
To complicate things even further, I add a creature's AC bonus in layers, with the base 10 being on the inside, then adding natural armor, then armor, then shield, then deflection or other effects. I add Dex bonuses at the end only to be surpassed by any Dodge bonuses. This means that if you actually hit cover on a ranged attack, the cover's AC (and Dex and Dodge) comes into play. If you beat the cover's AC, then it is a normal hit on the cover, but if it is a "hit" within the cover's Dodge and Dex bonuses, then the cover, if able, can elect to either to take the hit, or to have the attack miss them. This in turn means that the cover was NOT struck, and the attack slides through to the original target with its cover AC reduced by the Dodge and Dex bonuses, up to the original +4 cover!
In the example above, "C1" has an AC of 19, is Dodging the shooter, and has a +2 Dex modifier. If the shooter rolls a 21 against the target, he hits C1 as cover, as C1's AC is a 20 vs. the shooter (19+1 for Dodge). If the shooter rolls an 18, 19, or 20 though, C1 is given the choice to avoid the attack. If C1 does elect to avoid, the the attack moves to "C". Let say that C has an AC of 22, with a Dex bonus of +5. Now, if or when C1 elects to avoid shots on attacks from 19-21, C can also elect to avoid these shots because they fall within his Dex bonus. On rolls of 20 down to 18, C can decide whether he takes the hit or not. If he chooses to take the hit as cover, he blocks the shot from hitting the target, but still takes no damage as it wouldn't surpass his AC. If on the other hand C elects to avoid on those rolls (18-20), then the original target is still hit!
Basically, If our shooter attacks with something up to an 17, he misses everything, on a 18-20, hits the original target (if C1 first, and then C, both elect to dodge), hits C1 on a 21, hits the target on a 22 (if C elects to dodge), hits C on a 23-25, and hits the original target on a 26 and higher.
Ahh, archery. Such a joy to behold.
Aluvial