What do you mean by "didn't work"? The worst that can be said is that the Polyhedron contents didn't especially appeal to Dungeon purchasers and vice versa. The stuff in Polyhedron was usually top-notch, and I'd have gladly paid for a "Best of Paizo Polyhedron" hardcover (though I'm less inclinded to do so now that 4e is out).
"Didn't work" = "highly pissed off a significant number of Dungeon subscribers", from what I remember.
Originally, the RPGA was a paid-membership organization, and part of what you got for your membership fee was a subscription to Polyhedron.
The RPGA went away from paid memberships in 2002 (I still have the Chessex Battlemat, with an RPGA logo, that I was given in 2002 as a way to "pay off" my membership fee when they went to free memberships, and stopped publishing Poly).
Originally, Poly was primarly about the RPGA, with details about the campaigns, club competitions, etc.
The second generation of Poly (the Dungeon / Poly combo) had very little RPGA content, and really didn't have much in common with the original Poly. Paizo tried it as a way to save Dungeon, which, at that time, had sinking circulation, and was in danger of being cancelled. They felt that adding some d20 / non-D&D content to the magazine (via the "Poly" side) would increase the number of people who would buy it. Instead, they found out that Dungeon loyalists / D&D fans had little interest in the d20 content, and they complained very loudly about it. (I'd be curious to see some Paizo folks' POV on it, but it seems to me that it was the introduction of the Adventure Path concept that wound up being what saved Dungeon.)