In Battletech, 'Mechs serve a number of purposes:
1) They are all-terrain, in the broadest sense of the word. Unlike a tank or a hovercraft, 'Mechs can go into very mountainous terrain.
2) They are more maneuverable than traditional armor. Armor works just like tanks and cars do now. They move forward, and you adjust the speed of the treads or turn the front wheels in order to turn. A 'Mech can maneuver much like a human can, juking from side to side, making abrupt changes in direction, especially when coupled with jump jets that can fire laterally.
3) They affect both side psychologically. For your side, they are a great, immediately recognizable rallying symbol. That view of a friendly Atlas remaining standing while the smoke from entire salvos of missiles and autocannons dissipates can hearten any soldier who sees it. On the other hand, that same Atlas remaining standing will dishearten the enemy side. There's a reason the Atlas' head is shaped like a skull.
4) They represent a sort of "noble warrior" like a knight. Mechwarriors are incredibly popular on Solaris VII, and most people who own 'Mechs are fairly rich. This also feeds into the psychological side of the world--not necessarily combat, but life in general. The Mechwarrior is the ultimate soldier. He represents the best the Draconis Combine/Federated Suns/whatever other Successor State has to offer. A dispossessed mechwarrior is scarcely a warrior at all.
In a show like Gundam, they serve a similar purpose psychologically. In the first series, people feared a red Zaku. The Zeon troops couldn't care less about the core fighters, but they feared the Gundam. Another reason is ease of mission profiling. Since mobile suits have hands, they can use weapons that are shaped like normal humans would use. Swapping weapons requires nothing more than dropping the current weapon and picking up a new weapon. The fact that a beam rifle operated exactly like a normal soldier's rifle, the bazooka operated exactly like a soldier's rocket launcher, and so on also reduced training time; it made operation of weapons intuitive, requiring minimal training.
The reason you see a lot of melee combat with beam sabers and heat weapons in Gundam is because that series also had the added problem of Minovsky particles. Since they made radar effectively useless in battle, all enemies had to be identified at visual range. Therefore, all battles took place within very close ranges. That made melee combat a common occurence (as opposed to Battletech, where melee combat was a rarity, and only a few mechs, like the Hatchetman, Axman and No-dachi could truly engage in meaningful melee combat.
Speaking specifically to the Veritechs from Robotech/Macross (and the alpha and beta fighters from Mospaeda), they work much like the Zeta Gundam. They can use their fighter modes for speed, which allowed for interception of enemies at range. Their battloid modes provided the most firepower, using the same concept as the mobile suit. The guardian mode supposedly combined some of the speed of the fighter with some of the firepower of the battloid.
It should be noted that Battletech actually had transformable 'mechs early on, having used a number of designs from Macross. Those designs were later scrapped, and they were considered (in the fiction) too expensive to maintain because of their transformation mechanisms.
In Macross, though, the veritech is created using alien technology, and it's quite possible that such a design does not compromise structural integrity. It's the sci-fi version of "a wizard did it."
You're not going to find a real hard sci-fi reason to have mecha of any kind bigger than, say, the mecha in Avatar, the exosuits in Exo-Squad, or the one used by Ripley in Aliens. The real reason mecha exist is because they are awesome. They look vaguely human, and it's easier to see a mecha destroyed than it is to see a human killed.
edit: I'll add that until the Mackie, mechs in the Battletech universe were for construction, cargo loading, and so on. After the Word of Blake Jihad, when the HPG network went down, a lot of Battlemechs were destroyed or deactivated, and those mechwarriors found work using Agromechs and Constructionmechs. Those would later be repurposed for war, as well.
