It's not baked into the class because 1/3 of Warlocks use their Pact weapon for at-will damage, not eldritch blast. One of the worst things about 5e was that before the hexblade was released (which was basically an errata for Pact of the Blade, and caused more than its share of problems), even with the most optimized Pact of the Blade Warlock would still do more damage with Eldritch Blast than their Pact Blade. Pact of the blade was literally a trap, even if in you spend all of your character resources optimize your blade, Eldritch Blast was always the better option.