I'm not sure of the provenance of 'crunch', but it may be (at least in part) from Robin Laws's 'crunchy bits'. I don't much like the term myself, but it's not objectionable.
'Fluff' to mean 'information about fictional worlds, characters and situations' is objectionable because:
It's undescriptive. Such material is not flimsy, feathery, ephemeral, insubstantial, inconsequential, effortlessly changeable -- it's the substance of the fantasy worlds and games we're talking about.
It affects discussion negatively by casting such material as ephemeral (and the other senses of the word, such as to err, have no more positive connotations). It has this effect even when used by people who don't mean it that way -- it's just linguistically naive not to realize that words affect discourse.
It doesn't have a generally agreed meaning. I've read many online discussions where people have used it in different senses: with more or less derogative connotation, to mean 'fan fiction', to mean certain kinds of background rather than others... It arose in situations where setting and character was clearly subservient to mechanics, such as in parts of the Warhammer community, where it's still used in that sense.
It's silly and demeaning, because logically it frames everyone who ever wrote or imagined anything without game mechanics as a dealer in fluff. To see how silly and demeaning it is, all you have to do is imagine going to your favourite novelist and telling them how much you love their latest fluff.
It paints the communities where some people use it in a bad light to those outside. What kind of people call non-game-mechanical things 'fluff', this person wonders, and damage is done however often we reassure them it's not derogatory.