When does a rule violate copyright/d20 license?

[S'mon]

I know no court would let me go free with my own "Brain Flayer"

LOL

In the real world, unless you were actually copying WoTC's Monster Manual text about Mindflayers I think you'd have a very good case under copyright law, idea/expression dichotomy et al. The idea of a brain-sucking squid-headed monster is Lovecraft's not WotC's.
Potential breach of the OGL contract is another matter though.

 

[Ranger REG]

As long you do not use Wizards' expression of a brain-sucking squid-headed monster, go ahead.

Of course, anyone who is not well-versed in Lovecraft (only a small fraction of those in Wizards' customer fan base) may screamed "ripoff!"