Hmmm.... This might actually still be possible with Crossbow Expert. I would need to see the wording on it, but I see your point.
The nick attack with the scimitar would not have your dex bonus if you only had XBE but not TWF.
Eh. That's a lot of niche dropping of weapons, which you then have to go and pick up, for a few points of damage (3 or 4 on average?)
Or keep extras on hand, or just pick them up off the ground as your free interaction.
Also, I think I can do one better by starting with a Greataxe instead of a Greatsword and throwing a Cleave in there as well in addition to the other 4 attacks.