No, not everyone is affected at the same rate.
There was a campaign I played in from 1st to 13th level (a ranger throuout). We went from low magic to high magic to low magic again (due to time travel). There were obvious differences.
Our most powerful party member was a cleric 11/barbarian 2 - the player was experienced enough to know when to buff (so he greatly outperformed the straight cleric played by a much less experienced player).
Spellcasters can replicate the effects of many magic items for short periods of time. Magic Vestment and Greater Magic Weapon are some of the best buffs in the game under low item settings, whereas in high item settings they're often almost useless since they don't stack well. Spells like Mage Armor become more useful: the +4 AC doesn't stack with a +8 Bracers of Armor, but now it's +4 AC compared to nothing. Given only a round or two, spellcasters will be much more powerful than non-spellcasters. They also get an effective +2 DC boost out of it (cloak of resistance +5 vs stat boost +6 = +3 DC), although that advantage is lower at lower levels.
A high level fighter might do (using a one-handed +5 flaming bastard sword, starting Strength 15, all stat bonuses to Strength, no tomes, and a +6 stat-boosting item) 1d10+17 + 1d6 damage per hit (average 26). A non-magic fighter would do 1d10+9 (average 14.5), a bit more than half the damage. The damage of spells isn't reduced at all (indeed, a slight increase due to worse saves). Damage of both kinds benefit from the lower hit points of not having a Con-boosting item.