Again, where are you going to 'pencil in' this number? In a box on your sheet called "effective dexterity" or something? Now how many numbers am I remembering? It isn't 10 (4 stats plus 4 defenses), it is now 15! Again this is because you WILL HAVE STATIC MODIFIERS to your defenses. WILL HAVE. This is inevitable.
Now, lets imagine what you would have if you DID NOT have static modifiers. Instead you'd have to have modifiers to your ability scores directly. Every single effect would have to directly change them. You'd STILL have to write this adjusted number down, but NOW all the things that are derived from it would ALSO change!!!
There's simply no way that this change is an improvement. It might SEEM like one to you now without a lot of reflection, but believe me, I did the exercise of actually sitting down and saying "OK, suppose the rules look like this, what does my sheet look like, and how does this play?" and the answer is it makes things more complicated to 'remove' the 3 stat-based defenses. I know that might seem counter-intuitive to people, but ACTUALLY SIT DOWN AND INVENT A CHARACTER and you will see. Imagine this character with bonuses and penalties and items and etc on his/her sheet and you'll very quickly see that 'eliminating defenses' is the 'missing number falacy'. Just because you remove a number from your sheet does not make that number go away.