Here is a set of simple changes that can be used for a low magic world but maintain the power balance of standard D&D -- i.e. you don't have to scrap the CR system or worry about AC and BaB progressions growing out of sync as characters advance.
The first three suggestions compensate for the lack of magical armor and weapon plus bonuses as well as stat increasing gear. With these two things in place you get characters that are almost as powerful as standard D&D characters with typical gear (I've compared to the NPCs in the DMG and it works out rather nicely across all levels and classes) with the benefit that a character is defined by his own legendary ability and talent, not by his collection of phat lewt.
Note that these suggestions are for a low magic world that still has larger than life heroes in the tradition of stories such as Conan. Powerful magic exists, but is rare.
- All classes have a competence bonus to armor class equivalent to half of their BaB (round down)
- All characters get a stat boost every 2 levels instead of every 4.
- All damage resistance based on plus bonuses can be penetrated by any magic weapon (thus DR +1/10 is the same as DR +4/10 -- both just require a magical weapon to penetrate)
- Use variant paladin, ranger, and bard classes that get additional mundane abilities instead of spells.
- Spell casters are rare for campaign setting reasons (perhaps they are reviled and persecuted or maybe they all belong to very exclusive and elitist traditions or maybe magic is tied to specific bloodlines that are slowly dying out), but when they exist they are still powerful as per standard D&D rules.
- Only expendable magic items can be crafted by players, such as potions, scrolls, and wands.
- Magic items can not be bought except under extremely unusual and exceptional circumstances.
Some additional possible options, depending on what flavor you want:
- Magic swords and armor do not usually have plus bonuses. Such bonuses are no more common than other abilities, like flaming, ghost touch, and fortification. They are also all very rare, legendary, unique, and worth a king's ransom.
- All raise dead and ressurection spells are gone. Miracle can raise someone from the dead as per the raise dead spell, but only under very extraordinary circumstances and by will of a deity.
- High quality mundane weapons can have enhanced capability, such as a higher crit range, higher crit multiple, increased damage die, bonus to hit and damage, etc. to represent their extremely high level of craftsmanship. Such weapons are made only by the most famous weapon crafters and are very expensive (as per magic item costs in the DMG).
- If you want magic to be more low powered as well as more rare, make some changes to the spell casting classes to weaken their casting ability. Some suggestions have already been provided in previous posts.