Sure, it's also the smallest increment of falling damage.
A shortsword or a scimitar is a light finesse weapon, and a spear is a simple weapon that almost any character can use at level 1. As an improvised weapon, a shield lacks any weapon properties other than that damage die, and it will require a feat in order to really use it effectively. If you think about it, the shield is several pounds of metal and wood that has a broad surface you can really put your whole body weight behind as well as a narrow edge to focus the power of your strike on a particular point. It is a big, brute force bludgeon that requires a significant investment from the character to be proficient with, so I don't think 1d4 is adequate.