I'd be inclined to rule that the Command to "flee" wouldn't work to force a creature through spiked growth, because of the damage. I suspect that generally goes against RAI for this one.
But I also wouldn't necessarily rule that the spell simply fails to trigger, because "flee" doesn't say much about how exactly to do that. It just says "by the fastest means available." If dangerous terrain blocks the obvious way, then the target will try the next fastest means available by trying other possibly methods, like flying, jumping, or burrowing; or using a spell or item; hunting for a hidden door; etc. It's a compulsion, after all. So imo this situation really depends on the specifics of the creature, encounter location, etc.
tl;dr: I might have the target burn its round trying to flee (even if it moves zero distance), but it wouldn't do anything that would cause itself harm.