I don't know. I like the idea that some weapons are harder to use than others.
There are other ways it could be done though. I'm just not sure how complex things would get, and whether its worth it.
Minimum ability scores would do the job. It would make sense, since a weak person wielding a greatsword would get tired quickly and wouldn't do a very good job. And a clumsy person would be screwed trying to use a knife in melee against someone holding an axe. This would let a rogue be a thug with a huge axe, if he put the points necessary into his strength so that he could use it. Meanwhile, physically weaker rogues who's points went into dexterity or intelligence would just use smaller weapons.
Making class abilities dependent on weapon choice would do it. We have a bit of this with arcane spell failure, the need for a free hand for somatic and material components, and armor penalties on skills. We could add to that. If sneak attack were only usable with a small-ish weapon, rogues would suddenly be more eager to use that shortsword. And while people might want to play thuggish rogues who bash other people in the head with big axes, they'd just have to select different character options than sneak attack, or go with a fighter class.
Frankly, I think that the current proficiency system is the most elegant. If you want a straight rogue who uses a huge axe, take a feat. Or multiclass one level into fighter, gain some extra hit points and a point of BAB, a feat, and the ability to use an axe. While its true that its difficult to create the archetype of the thuggish axe wielding rogue at level 1, well, that's true of a lot of archetypes. Its easier to build them once you're higher level and have more blocks to build with.