This is not a judgement on either of you, but I believe that you both need to see a therapist, minister, someone who will be professional enough to help you work thru this.
My son's ex-wife had this attitude, and they divorced after 18 months, this being one of the main issues.