Of course! Just because something is fantasy doesn't mean there isroom at the table for everything. You may have different limits than I do, but we all have limits as to how far our minds will be allowed to dream before we cannot participate realistically.
I'll try to give an example. I once played in a campaign where the setting was in someone's mind. In other words, the characters were essentially members of someone's dreams (or even daydreams). Each night the characters went to bed represented the daytime when the person was awake. When the person would go to sleep, the characters would wake up and participate in this person's dream. The problem was that anything the person dreamed was legitimate. The landscape changed to whatever the DM thought it should - representing a new dream. The laws of physics were subject to the "dreamer's mental state." The campaign was cool for about 2 or 3 sessions. But I quickly found myself loosing interest because it pulled me too far from my comfort zone in being able to suspend disbelief in reality. It wasn't the kind of game I enjoyed.
The monsters within the MM can have the same effect. I occasionally look at a monster and just shake my head. It pushes me too far for what I enjoy. It isn't like I say it doesn't belong in the game - or even that I can't imagine it. It just pushes me outside the type of fantasy that I enjoy.