The Game Is Over
So, the real answer is that it was conceived as a game balance tool and nowadays is mostly a legacy mechanic. But if you want to know in terms of the fiction why some classes have larger hit dice, it’s a combination of physical toughness and conditioning that comes with physical combat, and skill at reducing the impact of a strike that comes from training and experience. While the wizard spent his time studying spells, the fighter spent her time practicing getting hit by things.
Yeah, legacy indeed.
So what of the rogue who grew up on the tough streets, getting in fights and such, surviving both on physical toughness and conditioning as well as luck and sixth-sense? What of the cleric who served as a soldier before adventuring and was crusader of sorts, getting in battles as well as having the favor of his god to protect him?
Why should these classes have d8 and a fighter or other such have d10 or better?