Sometimes I think it's not whether you Fudge that's the issue, it's whether you get caught fudging that's the issue.
Lots of DMs at some time or another have altered an outcome or roll, to increase the fun at the table, lengthen or shorten a combat, or anything in between. However, I know from experience that tons of players dislike finding out the DM fudged, even if it was on their behalf. It can cheapen a victory, or feel like betrayal if it led to a loss. I've found that just like some posters have said, deciding whether or not the roll is needed in the first place is the better plan. Most players I've gamed with are more okay with being told something just succeeds, or fails, then if they find out you let them roll, but their roll result was ignored in favor of whatever you decided.
Like they say, all a player has control of is their own actions. When a DM gives the player the chance to make a roll, the player feels like they're making a choice. If they find out that the DM already has decided what would happen, or changed the result, they can feel like that choice was taken from them, or was a lie from the start.
So, in essence, there's a lot of reasons to Fudge or not to Fudge, but I think the most important thing is to make sure your players don't realize you're doing it.