Then how does it work? There is a necessary verisimilitude to magic for many games, and many gamers... <etc.>
This is true enough. However, it has no bearing whatsoever on a rules discussion. You wanna know how the mechanics of HiPS work at the table? Read the rules, apply them. You wanna know how HiPS works in the gaming world? Make something up, the (game) world is your canvas. Please don't mix the two, the result is never satisfactory IME.
HiPS does not enable the character to go undetected - it enables him to use stealth. Will stealth work while the character is clanging a hammer loudly on his shield and singing out of tune tavern songs, simply because he has HiPS? I'd say no - stealth allows you to be silent and undetected. You are not being silent. If an enemy has hearing so precise your heartbeat is like pounding drums to him, your stealth cannot enable you to avoid detection by that enemy. And if your enemy can see through shadows and darkness, then the use of shadows and darkness to hide is reasonably ruled to be ineffective.
Bull. Sorry to be so direct, but what makes you undetectable when you use Stealth (without HiPS!) is not a lack of clanging a hammer loudly on your shield, nor is it a lack of Darkvision on an enemy's part (with HiPS). What makes you undetectable is the simple fact that your Stealth check beat everybody else's Perception check. The HiPS ability modifies the conditions under which you can use Stealth at all. The darkvision ability modifies your enemy's conditions under which he can use Perception, and how well he can use it.
Clanging a hammer might be an example of ex-post-explanation of why a Stealth check failed really badly. Similarly, the enemy hearing your heartbeat might be an ex-post-explanation of his extremely high roll on Perception, coupled with a good skill mod. But don't turn the whole thing on its head.
Again, my preference is a game with some verisimilitude, where special abilities work for a reason.
Then make something up to fit your bill while staying within the rules. It's not really difficult most of the time (ridiculous TO stuff like the Jumplomancer excepted, of course). But when you do it the other way round, you're not using D&D's ruleset anymore. You're making up houserules based on what you see fit. Which can be fine, but can also needlessly gimp players or playstyles, create imbalance where none existed before, or lead to more complicated adjudicatin down the road. Not that this is necessarily the case, just sayin'.
I read the Shadowdancer as manipulating nearby shadows to provide the concealment required for stealth, consistent with the statement that "they weave together the shadows to become half-seen artists of deception". The same HiPS ability could be explained by bizarre chameleon-like powers that enable the user to blend with his environment (an ability which likely also suffers when the viewer only sees in B&W, but maybe not - orange tigers blend with green foliage when the viewer sees in B&W), or by some quasi-invisibility bending of light around the user. Neither of these, however, seem consistent with the Shadowdancer's description as "A mysterious adventurer who walks the boundaries between the real world and the realm of shadows, and who can command shadows to do her bidding."
You can read the ability whichever way you like. But there's a rules-y way to read it, and that one's pretty clear-cut where the actual game mechanics are concerned. Adding stuff based on fluff is coming up with houserules (and in this case, gimping a not-so-strong class further).
A character is more than a bunch of game mechanics jumbled together with no rhyme or reason. The abilities have a cause and effect. Those should mesh with other rules of the game. Prestige classes, especially, should fit the game world and add to its backdrop. If they're just a set of mechanical attributes with no real tie to the game world, then they don't fit and should be removed from that game.
I repeat, make something up. If you really like a PrC's mechanic but not its fluff, change the fluff and keep the PrC. If a game mechanic has no rhyme or reason, change the rhyme or reason, not the mechanic. Fluff is mutable in a magical world far removed from real-world physics, which moreover makes heavy use of literary and pop culture tropes, and also wants the players to feel like heroes.
As well, nothing in the HiPS descrpition actually says it cannot be used in total darkness.
Nothing says it can. There's no rules for what you can't do when your dead, either. The one principle that all D&D rules are founded upon is that they must positively say what is possible. Everything not mentioned is impossible.
That said, in the case of the HiPS ability I agree with you that an area of total darkness should suffice to use it.
Since rangers have HiPS (Ex), is he using shadowstuff to make him invisible? Answer: no. I know the difference between HiPS (Ex) vs. (Su), but it isn't completely non-magic for one, extreme magic for the other, yet called the same thing.
I have adressed the Ranger HiPS (Ex) ability upthread. The fact that it's called the same as the Shadowdancer ability is bad editing on the designers' part. The two abilities work very differently, with the Shadowdancer version making many stipulations that the Ranger version doesn't. As I said before, a Ranger still needs cover or concealment to hide behind - and if he tries to hide from a creature with darkvision in an area of dim light, he will auto-fail, which the Shadowdancer will not.