A particularly difficult piece of terrain (especially one that causes damage or conditions) should be considered a hazard, and should have an associated level and experience (see traps and hazards in the DMG for examples).
However, advantages like cover, rough terrain, and elevation are fairly trivial, since they can work for or against both sides, and would not warrent any adjustment to the encounter.