The TWF rules even specifically list thrown weapons. And you can fire two crossbows, one from each hand, as explained in the equipment section under crossbow.
Yes, you just add up the penalties as usual and Rapid Shot adds one extra attack with your primary weapon.
Therefore, if the off-hand weapon is light, you have three attacks at BAB-4 each, one of those is with the off-hand, and two with the main hand (important for Str bonus for thrown weapons, for example).
Bye
Thanee