Try rolling stats in order (using whatever system you want: 3d6, 4d6k3, whatever). Now, this is random and gives the player no control. But after rolling six ability scores in order, pick two and swap them over!
Been there; done that. I'd have the t-shirt if they gave out t-shirts for that sort of thing.
You've address some of the issue of control in randomness, but you haven't addressed the fundamental problem that your system is equivalent to the DM assigning different players different amounts of points to spend before play even begins. You've not addressed the problem that in an average group of 6 players, using a method like 4d6k3, you'll have one player with 2 16+'s and nothing below 11, and another player with nothing above 14 and multiple scores below 10. So you've not addressed the fundamental problem that over the long run, the characters that are played tend to be the ones that are above average, and engaging in ever elaborate dice rolling rituals tends to drive that average expectation upwards.
Looking at the array you rolled, and mulling over the possibilities, gives you ideas that you cannot get in a system where you had total control from the start.
Usually this is only true in the sense that if you are making a wizard, you wouldn't normally have any points left over to spend in Charisma, and if you did you wouldn't spend them there but on something more reliably useful like Dexterity or Constitution. I think you are romanticizing how this plays out in practice, by remembering only the rare cases where it works out, and not remembering all the dull and uninteresting cases where it didn't. In the past when this topic has come up, I've illustrated this by rolling up 30 or 40 4d6k3 stat arrays to show how dysfunctional it usually is, but I don't have time at the moment. I leave it as a task for the interested student.
Point-buy allows you to metaphorically make your PC more stupid so that he is stronger. If you want a PC with high Str and aren't bothered with Int, why would you ever choose to have Str 14 and Int 10 when you could have Str 15 and Int 8, knowing that you have two racial +1 bonuses to assign? Thus, the evolutionary pressure for a population of 16s and 8s, with a dearth of (statistically more common) 10s and 11s.
This sort of argument actually cuts against your earlier argument that randomly rolled results were more interesting. In fact, a good many results will be 6 scores of between 10 and 14, with no obvious way to make the character in any way surprising or interesting. If you are really committed to random, you have to accept those pretty common 13,12,12,11,11,10 arrays as really interesting and inspiring results. Swap two out to customize! Exciting right!
Using this 'roll in order, swap one pair' method, your wizard might be strong or charismatic or whatever.
Yes, sometimes. But generally speaking, this only happens when you have a relative abundance of good scores. Once you give a player control, they are going to start optimizing. So the swap is going to be between something they really need, and something they really don't. If they don't have the needs well covered, but strength is high or charisma is high, then that salient stat is going to get swapped with the low Dex or low Con that they don't feel they can endure. In practice, once you allow any manner of control, these unexpected characters become really statistically rare except when they are also well above average characters.
Which is why in my own history of dice rolling char gen methods, I dumbed both swap methods I tried. I then tried not allowing swaps at all, but allowing the player to choose one ability that he could roll 5d6k3 for. But this still failed to work because even 5d6k3 is highly random and produces if not a large number of low rolls, still quite a few average rolls.
I imagine a university for student wizards. Each is likely to have high intelligence or they wouldn't have passed the exam to get in! But can we really say anything about what their other stats would be? Some people are charismatic or strong or what-have-you, and these people are randomly scattered among the population, including that university for wizards! This method much more accurately models that likely student population, because you can make sure that your Int is good enough by swapping a good roll into Int if it isn't good enough already, but four out of five of the others will be random.
But what about that same university using point-buy? What, every single one has Str 8 and Int 16? Really? Have you been to a university? Was every single student on the same course equally weak? Equally un-charismatic?
This argument would only apply to a university where everyone was a PC. In my own game, it would be rare to encounter any NPCs nearly as broadly gifted (as much points to spend) as the PCs, and whether gifted or not, they'd tend not to be optimized as PCs would be. So yeah, as a DM I'd be empowered to move useful high DEX or useful high CON out of those slots and into STR or CHR or WIS, in so much as it pertained to NPCs. As such, the reason for your loathing seems pretty strange to me. PCs are unusually gifted persons with unusually perfect aptitude for their chosen professions. NPCs however are not.
And in general, I haven't had as much problem with players choosing to perfectly optimize as you'd expect. It's very rare to see either perfect system mastery, combined with an aesthetic of play that considered survivability/power to be the be all and end all of chargen.
