Historically, there was a period during the 8th and 9th century when castles started popping up left and right all over France and Germany. They were built by minor nobles using only whatever local peasants could be forced into labor. Since these castles were constructed in defiance of the central government (such as it was), they had to be completed before the king could find out about it and stop the construction. Call it a few seasons work. Once the castle was built, of course, it is much more difficult to take them over and hardly ever worth the effort.
So, from historical sources, using only unskilled labor and no magic a small castle can be built in probably less than a year. A large castle (with ramparts and defenses and all that) would probably take several times that. However, if you use skilled engineers, a large, well trained, and disciplined work force, and above all magical assistance, you might be able to complete a large castle in less than six months. A Roman legion set up basic fortifications at the end of every days march and within a week could turn it into a semi-perminant structure with buildings, earthen walls, and trenches. Again, using no magic and only Roman era technology. If you had, say, teleportation circles from the rock quarries to the build site to cut down on travel time, a few summoned earth elementals to do the heavy lifting, and a really well thought out plan, you could have a large castle and enclosing wall for a medium town done in about six months I think. You would probably need at least 300-500 workers to build it, so their living quarters form the first part of the town. When people learn of a walled structure that may be safe from the denzians that inhabit the D&D world they will probably fill up the town within a few years. If not, you can always inport slaves and free them. Moving from town to city is more difficult, and will only happen if you have something more going for you than a castle. You would have to have an important port or be on a vital trade rout or be the capital of a thriving empire, or have abundant natural resources nearby to accomplish a population much greater than 4000.
So, from historical sources, using only unskilled labor and no magic a small castle can be built in probably less than a year. A large castle (with ramparts and defenses and all that) would probably take several times that. However, if you use skilled engineers, a large, well trained, and disciplined work force, and above all magical assistance, you might be able to complete a large castle in less than six months. A Roman legion set up basic fortifications at the end of every days march and within a week could turn it into a semi-perminant structure with buildings, earthen walls, and trenches. Again, using no magic and only Roman era technology. If you had, say, teleportation circles from the rock quarries to the build site to cut down on travel time, a few summoned earth elementals to do the heavy lifting, and a really well thought out plan, you could have a large castle and enclosing wall for a medium town done in about six months I think. You would probably need at least 300-500 workers to build it, so their living quarters form the first part of the town. When people learn of a walled structure that may be safe from the denzians that inhabit the D&D world they will probably fill up the town within a few years. If not, you can always inport slaves and free them. Moving from town to city is more difficult, and will only happen if you have something more going for you than a castle. You would have to have an important port or be on a vital trade rout or be the capital of a thriving empire, or have abundant natural resources nearby to accomplish a population much greater than 4000.
