I have two proposals...the first I am using in my campaign. . It is clear that there is a "magical" source for Forsaker's power, but they adamently hate magic, and dont see that they are benefitting from it. Thus, it could be a tenet of their tribe religion. They destroy magical items and slay mages in accordance with the teachings of their god, probably someone who once lived and taught them to hate magic in the first place, and whose teachings have been handed down through the generations. IMC, it is represented by an idol that they worship.
Said deity has his own special reason for these teachings, however. He feeds off the energies of the items and mages that they destroy, and grants them power in return. They dont even necessarily know that this is a being of power, and they do not know the origion of their remarkable abilities. Thus, they only THINK that they are entirely forsaking magic.
2. There are capabilities that humans had in the past that were somehow supressed by the use of magic. By forsaking magic, they can reawaken these latent powers.