BTW, I hope you're talking about 4e...
The DMG says a monster's level represents a fair (standard) challenge for a party of 5.
Five Standard monsters or a Solo of the party's level, yes. Note that a 'standard' fight is not supposed to be all that hard, it's outcome is not supposed to be in doubt - the PCs will win, they might win while expending few resources (just encounters and a surge or two), or they might feel the need to pull out a big gun, or take more damage than they might have hoped.
An 'easy' fight (a couple levels below the party) isn't even the speed bump a standard fight is. A party might even be able to skip the usual short rest after an easy fight.
1) I'd like to know how I can calculate its challenge if the party has a different number of people (say 3), but preferably based on solid rules.
The DMG has those rules. I know the table doesn't have parties of 3 on it, but it's trivially easy to derive it. Just note the difference between party of 4 and party of 5, and subtract that from the party of 4.
1a) Also for the above situation, what would be a "hard" or "easy" challenge?
A hard challenge is say two to 4 levels higher than the party. An easy challenge is 2 or more levels lower.
One thing to take into acount, though, is that there has been some power inflation and some changes in design philosophy. A MM1 montster will probably not give it's full advertised level worth of challenge to an anything-goes party with all currently available material. A MM3 monster of the same nominal level might really maul them.
2) Also, the reverse. If I know my party size, and what monsters I want in an encounter, how can I find out how *many* of each type of monster do I need in that encounter (eg. is it better 2x lvl2 monsters + 1x lvl3 monster, or 2x lvl3 monsters)?
Look up the standard encounter exp in the DMG, and just add up the monsters' exp values until you hit or go over the standard encounter total. Be very careful of using monsters more than a few level difference from the party. While an encounter can have a 'level +4' experience total, it shouldn't be made up of level+4 monsters...