Well, it varies depending on what game you are referencing. You're asking a question which actually doesn't have a single answer.
The kind of smarmy, obvious-on-its-face answer is "You use die X for task Y because the rules say so." And that's actually a valid, reasonable answer.
However, there's usually more to it.
Looking at D&D as an example, you can make a few general statements about the dice.
1. You use a d20, generally speaking, to determine success of failure, e.g. did you hit, did you make a save.
2. You use pretty much all the other dice to determine degree of success, e.g. how much damage did I do, what is the bonus provided by my strength spell (in 2e and earlier). Rarely, non-d20 dice are rolled to determine whether something happens (e.g. once per hour, there is a 1 in 12 chance of a random encounter).
3. Any die may have the function of assigning a random result from a table, e.g. wandering monster charts, random weather tables, random dungeon generation, etc. In this case, rather than determining anything related to success, the dice simply pick one (or more) of several possible results.
Smaller dice generate a smaller range of results. Weapons that do less damage have smaller damage dice.
But wait! There's more!
There's a distinction to be made between rolling one die (such as 1d20), multiple dice (3d6) or multiple dice in a pool, which D&D really only uses in character generation (best 3 of 4d6).
A single die roll provides an equal probability of each result, so (f'rex) 1d20 gives 20 results of equal probability. This is good if you want a wide spread of possible results or if you want a wide spread of characters to have a chance of succeeding at some task (assuming modifiers for character skill don't range outside of about 12 points of each other or thereabouts). In other words, using 1d20 to determine success lets characters operate together even if one of them only has a +2 to the roll and the other has a +15. It lets you set target numbers that offer both of these characters a chance of success while still rewarding players who invest in the relevant skill set (combat, crafting, whatever).
Multiple dice (3d6, for instance) look similar- you get a result of 3 to 18, which is close to 1 to 20, right?- but the probabilities are significantly different. Since you have a 1 in 6 chance to roll a 6 on each die, you only hit an 18 if all three dice roll a 6- a 1 in 216 chance. This is VERY different than a 1 in 20 chance. Most of your rolls will be in the 8-13 range, so average tasks will be easy for anyone above average, while difficult tasks will be very difficult or almost impossible for characters with lower stats, skills or modifiers.
Dice pool systems, such as in the World of Darkness game, let you roll multiple dice, but work differently. Each die that hits your target number or above is one success; generally, even one success is a marginal success, and the more dice hit the target number the better. Usually, rather than modifying your dice, a character's skills give you more dice to roll in such a system.
I'm not sure whether this really answers you question, but it's a pretty big question! I hope it at least starts to give you the answer you're looking for!