Sure, one wouldn't lose much in flavor by removing feat "taxes" and the like. But one of the trenchant observations of the character optimization community early on was that without fixing the inherent math problems of the game (hence, the "taxes" and the movement away from MM1 design to that of later books like the Monster Vault, etc.) and adding on to character combat efficacy by (a) hitting more often and (b) inflicting more damage with each hit, combat, one of the main features of 4E, moved from being a feature to a bug, via combat slog (combats lasting many more rounds than the 4 or so intended by the design team, lots of swings and misses on both sides, so none of those neat effects powers and feats and items carry go off, etc.).
Personally, I find it much more fun when combats are more "swingy," as enacted by this design philosophy in the later stages of the game, they are more exciting and the characters feel more heroic.
If the blandness of such feats is bothersome, simply give your characters two bonus feats (an Expertise Feat, early in the game, and Improved Defenses, later in Heroic or at the onset of Paragon tier) to fix the math and let them choose other, more flavorful feats.
If, [MENTION=82504]Garthanos[/MENTION], spending a feat on Superior Weapon training bothers you, allow a character to use a Background to grab a superior weapon, as well as military, etc., with Gritty Sergeant or Noble Bred for War. I don't think that would out of line with the benefit of Auspicious Birth.