You have it backwards: you divide the total monster "levels" by the total party levels. If the result is 1 or higher, then full ExP. If lower, the calculated ExP total gets reduced by the same ratio.
Oops. You're right. It is XP * (Monster levels)/(Party Levels)
Oddly enough, in this particular example the problem neatly solves itself.
Let's say (for ease of arithmetic) those Ogres each had 22 h.p. Appendix E shows an Ogre being worth 90 + 5/h.p., so these'd be 200 each; 3 of 'em makes 600.
Three characters take 'em down, each gets 200. 6 characters take 'em down, the ExP get divided more ways, so each gets only 100; i.e. precisely half of what each member of the smaller party got. I don't see the point in going through the motions just so I can reduce the 100 a bit further.
You're wrong: The three PCs get 200 XP each. The six PCs get 50 XP each.
That's not an insignificant change.
Cheers!