Back to the OP, though, it depends on how high of a level you are how much damage you expect to do.
For example, most of a rogue's damage is going to come out of their sneak attack. that's a flat +xd6, where x is dpendant upon your level. Taking one or two points away from an attack with sneak attack damage added on isn't going to be too noticeably different. It will still be less, but not overly noticable.
For fighters who don't plan on kicking in doors, grappling, or bull rushing, the same thing can be said. Power attack is still 2 for 1 on 2-handers. A buff spell on STR still makes you go up the same amount as a medium person's. A weapon enhancement is still going to add the same +x (or +xd6) to your damage. At higher levels, you'll feel your small size less so long as you stay away from being grappled, bull rushed, or -gulp- swallowed whole.
Plus, as a small character you can often get a riding dog and take it into the dungeon with you - whereas a medium character cannot take their horse.
There are advantages, don't let all the nay-sayers fool you. You just need to know the advantages and play them out while minimizing the negatives that have already been illustrated.
