Shoehorned is a loaded term. I believe not all of the 1,2, 3 AND 4e classes existed when the game was initially released, but I wouldn't call them "shoehorned" in. That implies they were shoved in to someplace that wasn't adequate to handle them, which I would say is inaccurate.
I do agree that a mechanic to allow for those type of actions would be nice, but I don't consider it mandatory. For instance, drinking a potion, or healing wouldn't be what I would consider swift/minor type actions. The game isn't all about combat, so just because something isn't as good as swinging a sword doesn't mean it's faster or negligible.
Opening a door or drawing/unhooking/picking up a weapon, sure.
Eh... "shoehorned" is a pretty accurate description of swift actions, although I've been using "bolted on" which I think is a bit more on the mark. Swift actions were the culmination of a lot of fudging and filling to handle stuff like quickened spells. They do the job, but the game was not originally built to incorporate them, and they have some odd quirks (e.g., you can convert your standard action to a move, but not to a swift). Then you add in immediate actions, free actions, not-an-actions... the action economy of 3E was a kludgy mess by the end.
IMO, the ideal situation would be if D&DN neither had nor needed anything except move-and-standard. But the developers need to do some heavy soul-searching to be sure they can go down this road and stay within that framework; not just in the core books but in the parade of splatbooks that follows. If we wind up with a lot of "you can do this once per turn in addition to your normal action," they'd have been better off to include minors from the start.