What I'm trying to say is that this isn't an inherent feature of D&D, but a design choice you are making. If you don't like it, don't choose to play that way. It's quite possible to look at D&D and go:
Level 1 = Gritty
Level 3 = Heroic
Level 5 = Paragon
Level 7 = Epic
You don't need fireballs to kill goblins. By the time fireballs show up, old school CR 1/3rd goblins in a straight up fight are no longer a problem. Your party can slaughter them scores at a time. A 3rd level wizard with burning hands, scorching rays, flaming sphere, and a wand of 'flame dart' (ray of frost, only firey) is a hugely scary fire wizard that can leave a trail of scorched goblin bodies in his wake.
The relationship of your character to authority, the Lord in your example, is also something you make as a choice. If the lord is a 1st level expert, with a handful of 1st level warriors with 11,11,11,10,10,10 stat blocks, and a valley full of 1st level commoners, then even the 1st level Wizard with elite stats is a valueable resource that can be called on to face those dastardly and dangerous goblins.
Early on in the D&D inspired Magician: Apprentice, the hero casts a fireball, and everyone is astounded, "That's amazing. What you just did is impossible. Surely this is the most powerful wizard in the whole world!" Everything is a matter of perspective. If the most powerful wizards in your world are legendary 5th level characters, there is no expectation of needing fireball to be important. It's only when 5th level characters are pathetic or baseline, that you feel pathetic without fireball.
Sure, but it doesn't take enormously powerful characters before a party of them can slaughter CR 1/2 orcs, especially if a little care is taken.
I guess that depends on what you think you need. In early 1e, 1st level characters were so fragile that you needed 1-2 more hit die to have any sort of immunity to bad luck at all. Then everyone started using the rule that you didnt' die until -10 and things got a bit better. By 3e, starting with max hit points, good con, the possibility of a toughness feat, and not dying until -10 hit points, you've got lots of protection from luck and everything you need to have fun.