Both. Just depends on where you are.
Closer to the surface, generally, things are darker. The luminescent fungi and glowing crystals and such like one might encounter in the deeper depths and more magical areas of the underworld tend to be some miles down, through the unlit bits. But closer to the surface they are rather rare.
You could, also, easily be down low and not find any naturally occurring or magically enhanced/placed -so, like, around certain species' settlements- that will provide some degree of light. It really is something of a crap shoot, and will change from corridor to cavern to grotto to passageway. And there may be places with lots of light and places with very little...even if the "natural" light source tropes are in the area.
Is there moss everywhere in a given forest? No. No, there isn't.
And yes, you will need to use a light source or have darkvision -or infravision- in those places.