You can't skip the damage, no. Actually the only reason you can skip pushing people is specific to forced movement - you can push/pull/slide people up to the amount specified, and the amount you pick can be zero. With other secondary effects like a daze or ongoing damage or something, you can't...