Yet again seconding the "play the bard to his strengths rather than just tossing him into combat" opinion. Playing foes "smart" allows you to get a lot more fun and use out of them, rather than just using them as throwaways.
Additionally, the CR system preassumes that you're giving your NPCs and PCs the standard wealth and magic as per the books. If you violate those assumptions, you're going to need to re-evaluate most of your CRs, not just for characters but for monsters, too. Low-level challenges like kobolds and orcs and the like won't be affected much, but when the party gets to higher levels, some monsters will be MUCH tougher than their listed CR would indicate. You're going to want to eyeball everything if you aren't doing so already.
Heck, I'm in a similar fix even WITH standard wealth and magic in my campaign because the CR system assumes something roughly equivalent to the "classic" fighter/mage/healer/rogue setup, rather than the caster-heavy party that I've wound up with. Evaluating each encounter against my party's capabilities is something I learned the hard way after an encounter of "moderate" difficulty unexpectedly almost turned into a TPK.
Good luck!
