To get the CR below one, I'd first recommend reducing it to 1 HD. That would reduce the BAB, grapple, attacks, and saves. You'd also have to eliminate 3 skill ranks (I'd take them off of Listen and Spot) and one feat (probably Blind-Fight, because without Weapon Finesse they'll never hit anything). Reduce spell resistance to 11. I think you can keep swarfighting. Upsizing it to Medium probably won't affect CR, but by the guidelines it should get +4 Str, -2 Dex, +2 Con, although you could fudge that a bit. If you do use these ability changes, ditch Weapon Finesse rather than Blind-Fight.