But then, you should be losing those hit points from your con score anyways, just as you would if you had your con drained.
What you could do is subtract the bonus HP from con, (1, 2, 3 hp per level, or whatever) and then roll the extra hp only. So that, if you previously had d8 HP, you would...