Opportunity cost - Wikipedia, the free encyclopedia is a economic term that references the cost of choosing one option against all possible options. For example, if your rich uncle leaves you $40,000 on your 18th birthday, you could spend that money on fast cars & loose women, or you could spend it on college. The opportunity cost the the intangible "cost" of choosing one of these options; college is hard work but it means more money in the future (by path of a good career) while fast cars and partying creates enjoyment now at the cost of long term gain.
Any choice carries an opportunity cost; the cost is what is "lost" by choosing X over Y.
Now, back to D&D. The opportunity cost of devoting two ranks to a skill you will otherwise not use (such as diplomacy) vs. spending those ranks on a skill you use all the time (say, spellcraft) is the frequency of use vs. surprise bonus.
If I make spellcraft checks regularly (to identify spells, to identify potions, etc) than that +2 (10% boost) is regularly used. Your much more likely to roll that skill and thus have that +2 come into play than a skill you do not use regularly or only use in "forced" situations (such as your "The King dislikes the bard and asks the cleric what he thinks").
Sure, that +2 is nice when it comes in handy, but you'll get more miles out of putting those 2 skill points into your spellcraft which you roll multiple times per game session.
Of course, D&D puts a bit of a crimp in that by making certain skills have earlier caps than others. Skills that face a static DC (such as climb, balance or tumble) often have "cut off" points where further ranks is wasted (such as no need to raise tumble over a net +24 bonus, synergy and ability mod included). Those "wasted" ranks often show up in vanity skills or to shore up other skill point oddities (multi-classing, int boosts).