I don't understand what you're saying here. What do you mean by "let's try a party of X?" Are you talking about the number of PCs? The number of goblins? The level of the PCs?
He means the number of goblins.
When I calculate using the official method, I get 14. I don't understand the magic that happened for you to get that down to 12. By my calculations, a party of 4 PCs against 12 goblins (outnumbered 3 to 1) is only (12)(50)(2) = 1200 XP. That falls well below the hill giant's 1800. Your 11-goblin team is even worse, at 1100 (by the official method) -- and that is generously saying that 11 vs. 4 is a 3:1 ratio. The total for 11 goblins should actually be slightly less.
The key is that you can't easily figure out the number of goblins that is equivalent to a giant in the official encounter XP multiplier system because you have to try out multiple numbers of goblins and see if the XP per goblin * the number of goblins * the multiplier equals the XP for the giant. You can't simply divide 1800 (the giant's XP) by 50 (the goblin's XP). And a group of 11-14 monsters is x3 XP, which is how he gets the answer of 12 goblins. 50 * 12 * 3= 1800.
Encounter XP Multipliers
Number of Monsters XP Multiplier
Single Monster —
Pair (2 monsters) × 1.5
Group (3-6 monsters) × 2
Gang (7-10 monsters) × 2.5
Mob (11-14 monsters) × 3
Horde (15 or more monsters) × 4