I agree that you need to take into account the composition of the party, in particular because one need to remember that the encounter calculator is based on standard characters with no options. So you need to take to heart the section about modifying encounter difficulty as well: "Increase the difficulty of the encounter by one step (from easy to medium, for example) if the characters have a drawback that their enemies don’t. Reduce the difficulty by one step if the characters have a benefit that their enemies don’t. Any additional benefit or drawback pushes the encounter one step in the appropriate direction. If the characters have both a benefit and a drawback, the two cancel each other out."
The examples given are situational, but often forgotten, but the same applies for example for a party with high stats (standard array is used as the base), options included (feats, multiclassing, basically everything that is used in optimised character creation), magic items (again the basis of 5e is that magic items, in particular those giving bonuses, are optional and not part of the base, etc.
We often see people complaining about the result of the encounter calculator and saying that it's too easy, but when asked, say that indeed their party have rolled their stats (and of course, "luckily", all have very high numbers), all options allowed, powerful magic items, etc.
Not to mention how many encounters you have, party makeup, general tactical astuteness. I was running two campaigns for different groups at one point using the same optional rules, roughly same amount and type of treasure, similar enemies and so on. Group A could simply handle far more than group B. By quite a bit. There's no formula that could take into consideration that one group could handle a target XP budget 30% higher than the other.