An interesting discussion.
To Firelance (and LostSoul): you've suggested to LostSoul that many things a DM has to use judgement for in a SC (what skill to use, outcomes, modifiers, etc.) should be based on what is logical, what flows naturally out of the situation and the game world. That's definitely part of the story, but not all of it. I think you should also pay some attention to mechanical considerations, the "fiddly bits" of the SC to be used.
Take selection of skills to be used. I think Mearls wrote in one of his early skill challenge articles that you should look at the skills your specific players have, and tailor your list to that. (to the extent that you have a hard and fast list of skills, which you might not, as discussed.) Not that every skill on the list must be one a PC has trained, but you should have an eye towards them.
Similarly, when it comes to setting DCs. If one PC has a monstrous skill bonus on a particular skill, you have to be aware of that. It can literally break a skill challenge if they can auto-succeed against the standard DC (and the players realize they are in a skill challenge, etc.). You'll probably at least want to cap the number of uses of that skill. To avoid punishing him or devaluing the choices he made to obtain that bonus, you can provide extra benefits on extreme successes, and/or optional, obviously extreme difficulty options where successes convey extra benefits, that the player can choose tackle and get a chance to shine (more Mearls suggestions).
We don't want to throw logic and verisimilitude away, by any means. But we have to balance them against what will make for a fun skill challenge mechanically. This is more art than science; really just a specific case of the general balancing act a good DM has to carry off.
If the only logical skills for a challenge are all untrained by the PCs, perhaps that is not a good challenge for your party.
Or, add to or change the situation, and/or the game world, so that your desired fiddly bits
do fit naturally. An example I used in discussion on WotC forums was a wilderness travel challenge, where the PCs were stuck in ruins deep in a desert and wanted to get back out to civilization. The obvious, logical choices for skills are probably Endurance, Perception, and Nature. In my group I think only one PC had Endurance and Nature trained, and no ability or other bonus on Nature at that. Now, it might be fine for a brief challenge to have him roll Endurance over and over while the others aid or roll Perception.
But what if I wanted something longer? OK, let's add something to the situation that will allow the scholarly mage character to participate. There are remnants of enchanted roads built by the lost empire buried under the desert. Now he can use Arcana to help navigate. What about the cleric? There is a curse on the ruined city that makes travelers walk in circles and inevitably return to the city. Religion becomes another usable skill - indeed, a critical one, since failure to gain a success each round is an auto failure, as the curse draws them back in. Will this seem "gamey" to the players? Perhaps... but it might just make the setting seem more alive and interesting, especially if you plant clues about the roads and the curse in the ruins for the players to discover before they set out.
Regarding another specific issue... I don't think a success in a skill challenge should always produce the exact outcome the player was shooting for. What's important is that the success move the party closer to the goal.
"I use Athletics to bust the door. "
"You're unable to break down the steel door. But your repeated battering has knocked off the accumulated dirt and dust, revealing a mystic inscription."
"I use Diplomacy to convince the king to give us the artifact."
"The king is impressed, but not fully convinced. What if you are unable to protect it from the demon? He asks for more proof of your prowess."
The reason why I bring this that I'm wondering how this interacts with skill challenges. If you want to get past the guard to get into the spice den so you can murder a smuggler and his shifter companion, resolving the single action or task in that manner will resolve the entire skill challenge - and that's something we don't want.
(OK, as suggested getting past the guard might not be an appropriate scope for a skill challenge, but for the sake of argument let's say it is.)
In this case, unless it's the final success in the challenge, success at telling the guard to go away will not cause the guard to go away immediately. Maybe it will make the guard say, "I'd love to head down to the pub, but the captain's gonna come by on inspection rounds any minute and I'd lose my job. I can't leave until my relief shows up." The success still moves the challenge along, presenting new avenues for the players to proceed. Now they can try a Bluff check - "Oh it's okay, I spoke to the captain, he said to take the night off." Or they can follow up with the captain, convince him to make his inspection immediately, after which they can leave. Or disguise themselves as the relief. Or use Streetwise to let him know about another employment opportunity with better pay and better hours, so who cares about this job?
But anyway, except in very exceptional cases, a single success shouldn't be able to short circuit a skill challenge. (and again, if it seems like it easily could, perhaps you should rethink the scope of the challenge.)