BAB doesn't cover everything though. It's unlikely a wizard will have the Strength to power the things fighters do, including Power Attack.
Let's compare that random UF fan to a wizard though. Said fan will probably be equivalent to 1st level, maybe 2nd commoner. 0 or 1 BAB is fitting, and an equal level wizard would be in the same boat. Neither is even proficient with unarmed strikes, so -4 attack right there and provoking AoO's. Neither is likely to be able to lift more than 100-130 pounds over their head (Strength 10-12). And they both have crap HP with d4.
The actual fighters though? They're proficient with their unarmed attacks, they've got Improved Unarmed Strike, and they've probably got Weapon Focus and Power Attack too. They're at least 3rd or 4th level with full BAB, and have d8 or better health. They have at least 14 Strength, going up to 18 for some of them.
Let's compare their attack bonuses: commoner and wizard are at -4 to -2. The UF guys are at +5 to +9. Take away the BAB from the wizard and he's still in the negatives. Put the wizard up to BAB +5 at 10th level and his attack in this case is only +2 at most unarmed without his magic, which is still only a third or fourth the capability of the fighter. For emphasis, even if the wizard is double the level of the fighter, he's still less than half as good at typical combat without his spells.
Combat is far more than BAB. The wizard is fine where it is, as are the other casters.