It occurred to me that I would probably need a assortment of custom, post apocalyptic style weapons (Like bat with nails in it, or some Fallout-inspired creations).
I would still use the current groups (simple, martial, exotic, etc) but each weapon in those categories are different. How do I decide their stats and hit die without making a single weapon OP?
So, there are a bundle of questions here, some with 'no wrong answer' provided you are consistant about it.
In general, for melee weapons, the D&D weapons list is going to be perfectly adequate. You just need to stat an improvised weapon as the melee weapon it most closely resembles. For example, a bat with nails through it is an improvised morning star.
The next question you need to resolve is whether an irmprovised morning star is actually as good as the real thing. That is to say, "What level of craftsmanship is involved in making a bat with nails through it?" In general, improvised weapons like that should probably have a penalty relative to high craftsmanship items like a purpose built morningstar created by a blacksmith with tools, a forge, and proper materials. In general, I'd suggest either a -2 penalty to hit or a -1 penalty to both hit and damage. The items would often also have reduced hardness and durability compared to purpose built items.
But, you might also decide that you'd like to encourage scrounging and improvising weapons as part of your game Fallout style - Craft is a very valuable skill in a low tech world without a functioning economy. In that case, you might argue that some of the improvised weapons are actually representative of high levels of craftsmanship despite their jury rigged appearance (or maybe even because of it). In that case, decide whether the bat with nails through it was made by a weaponsmith with good craftsmanship and so represents an improvised, normal or masterwork morningstar.
The next thing to decide is whether or not you are going to replace magic with science with respect to melee weapons. If you are, that means that are levels of normal craftsmanship that can be obtained that are above masterwork in quality. Maybe you want to have titanium clubs with hollow cores containing a mercury bead that increases the momentum of the club when it is swung, etc. In that case, you need to add a schedule of DC's and costs to manufacture such items and their statistic above the level of normal items. In this system, you'll also want to make 'scrounge' an important part of the treasure you can find - "Hey guys, look, these are sheets of pre-fall carbon fiber - we can use this to greatly reduce the costs of creating a suit of superior armor." This also may go back to your earlier question about special materials. If you don't have magic, then you can safely replace magic with special materials because you aren't 'stacking' the same bonuses.
The next thing you are going to need to decide is how you handle really high tech weapons. For high tech melee weapons - vibroswords, energy maces, nanofiber whips, lightsabers, etc. - this shouldn't be too hard since we already have some notion of the scale of what is balanced by looking at magic items. A vibrosword might simply be the equivalent of a +2 keen brutal longsword - or the equivalent of whatever represents superior craftsmanship for a longsword according to the DC's above plus some otherwise 'magical' enhancements above and beyond what normal superior Craft(weaponsmith) is capable of.
High tech ranged weapons - everything from muskets to molecular disrupters and energy blasters - are more problimatic, both because there aren't good rules from them in D&D/D20 (IMO) and because if you do introduce them they will greatly alter the balance fo your game even if only available in small numbers. You'll need to decide what tech level is commonly available and what items postfall craftsmen can reproduce, repair, and produce ammo for whether it's blackpowder muskets or AK-47's or RPG-7's. You'll then need stats for those items, and you'll want to add those items to your 'simple' weapons list. You'll then need stats for the 'magic' items like modern squad assault rifles, 20mm smart grenade launchers, laser weapons, sonic weapons, microwave emitters, molecular disrupters, kornet guided AT missiles, plasma cannons and whatever else you want to have for pre-fall tech. These are effectively 'charged' magic items that work and can be recharged only if the players find the scarce and irreplacible ammo needed for them to work. And you'll need to stat them out and add them either as simple, martial or exotic weapons as you deem appropriate.
In other words, you'll want rules for guns. And in some cases, these guns are going to be vastly more powerful than anything normally available in D&D to a low level character. Low level fighters with modern weaponry have effectively high CR.
The best rules for firearms out there are Kenneth Hoods. Find his firearms pdf if you can. I have a beta copy; I don't know if he actually published them at some point, but they are awesome and address all the problems I have with typical D20 rules for guns.
With his rules as a basis for what a 'balanced' gun looks like, you can work out what the superior tech version - lasers, blaster weapons, etc. - looks like.
