It turns out the galaxy is a very dangerous place. You'll find lots of exploratory expeditions in our history carried weapons - Magellan is known for demonstrating arms, armor, and artillery, for example.
I think the distinction comes in the idea that the primary purpose of a military organization is to engage in warfare. This distinguishes them from police or peacekeeping forces, which are also armed, but not generally intended to engage in warfare.
Admittedly, this brings into question what counts as "warfare". We can note that Starfleet isn't known for putting together large groups of ships to engage in violence, except in defensive actions like Wolf 359 and the Dominion War, which the Federation did not start. If they are not usually gathered in tactically relevant groups, it is hard to consider them military.