You CANNOT go with crossbow expert. Pact of the blade specifies melee weapons - and crossbows are ranged. So, no Extra Attack, no extra +Cha damage.Going blade pact with rapiers and dex 18 means you never get disadvantage on attack rolls in melee combat, dealing an average of 17 dmg at level 5 (two attacks), having a solid AC and probably more fun than standing in a corner firing eldritch blasts. Combined with feats like crossbow expert (or polearm master, if going the strg path) you even get one additional attack early. That increases the average dmg to 24,5.
As well, if you're using two handed weapons, you must keep in mind that spells, like Hex and Darkness, require you to mess around with material components, requiring a free hand. If you can prebuff before combat, then its not an issue. If you have to do it after or during combat? You're in trouble.