Crothian said:
I just don't see it. One does a point of damage more on average, the other crits 5% more (threat to crit anyway) and has can be used with weapon finess using Dex, the most powerful stat. The ability to finess a rapier is huge.
Looking at math, using similar abilities for both weapons:
A "plain" (no improved crit or keen) Rapier does 1d6+x damage on a hit, and 15% of hits (ignoring the cases where you won't hit on 18+) are crits that do double damage. So average damage per hit is 1.15(3.5+x), or 4.025 + 1.15x.
A plain longsword does 1d8+x damage, and 10% of hits are crits. So average damage is 1.10(4.5+x) = 4.95 + 1.10x.
So, how much of a bonus do you need to do the same average damage with both?
4.025+1.15x = 4.95+1.15x
0.05x = 0.875
x = 17.5
So, if your total bonuses to damage are 17 or less, a plain longsword is better than a plain rapier. Let's look at the case where you have one crit enhancement (improved crit OR keen).
Rapier does average (3.5+x)*1.30 = 4.55 + 1.30x
Longsword does average (4.5+x)*1.20 = 5.4 + 1.20x
Equalling them gives
4.55 + 1.30x = 5.4 + 1.20x
0.10x = 0.85
x = 8.5
And with a double crit enhancement (improved crit that stacks with keen), we get
Rapier does average (3.5+x)*1.45 = 5.075 + 1.45x
Longsword does average (4.5+x)*1.30 = 5.85 + 1.30x
5.075 + 1.45x = 5.85 + 1.30x
0.15x = 0.775
x = 5.17
Getting a damage bonus of +18 is pretty heavy-duty stuff. A damage bonus of +9 is doable, especially at the levels you get Improved Crit or feel like you should be splashing out on an ability like Keen - +4 for Strength, +2 for specialization, and then some things like prayer on top of that. A damage bonus of +6 is quite easy to get.
But that's not really showing the whole picture. In my first example, I ignored the extreme cases where you only hit on a 19 or 20, because those reduce the utility of the rapier compared to the longsword (because of wasted threat range). That's a good approximation, because you won't often be facing things that hard to hurt. However, in the more advanced examples, you have a threat range of 12+ on the rapier. It's not at all inconceivable that you could encounter monsters with an AC that requires rolls higher than that, especially on iterative attacks. So the damage for the rapier is slightly inflated on those levels. Plus, there are all those monsters who don't care about crits.