The non-dragon CR 20 monsters are the pit fiend, balor, and tarrasque, which have a +30, +33, and +57 attack bonus. That's not very hard to defend against. Your initial build means that the fiends need a natural 20 to hit you, and the tarrasque is not supposed to ever get an attack roll against...