The baby itself is a half-fey, which in my DM's world grow up quite significantly faster than human children, about twice as fast, and the DM does track things like travel time, so if we move around a lot (which seems like it might be the case), and nobody has teleport (which it doesn't look like will be the case, as the only person capable of getting it will be a sorcerer, so they might not pick it up), then travel time alone might make the game progress at a decent rate. The last game I played with this DM covered about the span of 2 years in game over about a 8 month period out of game, and that was a very confined game with little to no overland travel.
Finally, I have a petal familiar (using the familiar ACF) who could take care of the baby while I'm going on particularly dangerous adventures, hiding the child away in a tree hollow or the like and using the emphatic link to keep in touch and immediately retreat to the baby if the familiar senses something wrong.
If it starts to drag, a 'hidden' baby could always be abducted to the feywild (or other faerie equivalent realm). Time does funny things there, and the child might suddenly be a lot older.
It sounds like it could be a great idea, and there's always an out or three if needed.