Absolutely. That's been our rule here forever; and on the rare occasions a player takes a sheet home and forgets to bring it back next session, it's a given that said player will miss part of that session due to going home right then to get it.The character sheets stay with the GM, because the GM won't forget to bring all the notes to every game. If you want to look at your character sheet at home, take a photo.
When I said "keep the old [sheet]" I meant clip it to the back of the new one rather than crumple it up and throw it away, which I've seen done. (online or digitized sheets are even worse for this, as it's too easy to just edit what's there rather than create a duplicate file and edit that so as to preserve the old one).