You can still dual-wield if you want, it's just your bonus action attack is dependent on it. Also if its an unarmed attack, you still get Ability Modifier damage, so technically 12 rather than 2. But yes, you will take a damage dip if you stuck to punching and didn't pick up tavern brawler.
But...