Fairness to a Game
The only way to be "unfair" to a game, IMHO, is to present it in a completely different light than its creators intended. The way I see it, fairness, which is, ironically, a loaded concept, can only be judged impartially by comparing what actually happens with the obligations derived from social contracts. You can't have a social contract with a game, just as you can't have a social contract with a Stop sign -- a game is not the right kind of philosophical entity. But you can have a social contract with the game's creators, who have done this and that thing in order to provide a game experience which has these and those characteristics.
You're still playing D&D in the same general way even though, say, you allow clerics to take rogue paragon paths (I know, I know; bear with me). But you'd probably be pushing it if, say, you rolled a d30 every time the game calls for a d20. And you'd probably be violating the social contract with the game's creators if every creature was a minion, they all carried treasure many levels above their own, they were only encountered by characters a couple levels above that, and everyone was rolling d100s instead of d20s.
The game is what it is. The game designers have presented it just so, and I'd say deserve to have the game given a fair shake as is. Contract fulfilled. After that, if you find your experience of the game (not the game itself -- your experience of it) improves with some (or many) modifications (or complete overhaul), then go for it.
Fairness to Your Players
Is everyone having a good time? Are your players aware you have modified the game's rules?
If "yes" and "yes," you're being fair to your players.