Please note that hurling an object at high velocity is not the same as "on contact."
You can punch someone or throw a tree a them at high velocity, but you can also pat a hand or sit on a tree without your bones shattering.
Whereas fire tends to burn regardless of how fast it's smacking in to you. Which was my point.
Aside from fire, the four elements are just standard matter generally harmless matter. You have to make them move, generally, for them to be of any major threat. In short, Dirt is not a damage type. Unless you are a prom dress.
--
My vision of an elementalist, based on my own musings and inventions over the years -- they're a pretty huge deal to me -- is partly visible in the animated series "Avatar: The Last Air Bender." The "benders" are basically elementalists, with the main character being a "mage"-style elementalist, while the rest specialize.
More specifically, I see an elementalist being an extremely active shaper of the elements, molding them like clay, and being able to Sustain most of their abilities. A wizard, in general, just drops things on the ground, such as a Wall of Ice, and moves on to another spell. An elementalist summons forth a Wall of Ice, and has it chase people down like a frosty anaconda, and perhaps makes the wall of ice explode after it's corralled the entire battlefield in its grip.
While a wizard throws punches, an elementalist wrassles.
An elementalist would have fewer options than a wizard, but would be more flexible with those options, and would probably have less initial power, but would be able to use that power for more rounds.