Why do you say that? If you are using it as an improvised weapon then you are not wielding it as a shield, so you don't get the +2 AC.
The words "then you are not using it as a shield" are an assumption, therefore to rule as such, you would effectively be houseruling. In fact following the wording strictly for shields on 147 PHB "-WIELDING a shield increases your armor class by 2". As established, shields CAN be used as improvised weapons and can be a dual wielded weapon if it's wielder has the dual wielding feat. More importantly, any object you hold is CLASSED as an improvised weapon, does attacking with your shield stop you from wielding it? No, and according to the rules, it doesn't matter HOW you are wielding it, if you are wielding it, you get the AC bonus.
I'm not saying that the game designers wanted this to be allowed, I'm saying that strictly following the rules for improvised weapons to the letter means that a shield is classified as a weapon. If the shield is classed as a type of weapon, the dual wielding feat can apply to it, the same as if a player was holding a crowbar and a bin lid (though the bin lid would not be used as a shield unless the DM house ruled it).