I feel like it was worded this way because the assumption is that the PC is available since this is the downtime section of the book. Construction is something you do when you have nothing else going on. The PC not being there is supposed to be the edge case which the rule covers.
Wasn't there a thing somewhere that said that milestones generally granted 10 days of downtime? I think it was in the Adventurer's League packet. I will be interested to see what suggestions the DMG might have on how much downtime might be given in different circumstances. Some things, like the castle, would require a significant break from adventuring to complete such as a break of several years.
DM: Time passes
...
DM: You have a castle!
PC: Squee!