Let's take an example of what I am looking at: Giants. Most versions of giants describe that they can throw large boulders and gives rules for how much damage that can do, etc. Taking those rules at face value you might equate giants to catapults or other siege engines, which weren't commonly used to knock down walls.
Well, first, by the late middle ages torsion and gravity based siege weapon technology had advanced to the point that in point of fact if you had enough money you could knock down the walls of pretty much anything with a trebuchet. It was just a matter of time. But for the most part, this was nothing really knew. The whole point of a castle was to delay a large force with a small force until you could muster forces to relieve it. Still by the late middle ages, the presence of powerful siege engines like trebuchets forced castle designers to begin building castles with counter-batteries of siege weapons of their own to destroy attacking trebuchets and latter cannons.
Early cannons didn't really outperform trebuchets and existed alongside them for decades with proponents for each. Likewise, once cannons were around, castles were increasingly built with cannons to destroy attacking cannons.
The model of fantasy castles ought to be those of the late middle ages, and not that of earlier periods, simply because technology in fantasy games tends to be late medieval or later (sometimes much later).
But let's compare giants to normal humans for a moment and extrapolate from there. Any decent baseball pitcher can throw a ball at 90+ mph accurately at a target.
Well, to begin with, I'd argue that the ability to throw a fastball accurately at 90+ mph is rare even among baseball pitchers. If you can do that, you're pretty much guaranteed to get a job as a professional baseball pitcher for a considerable salary simply because the ability is so rare and in such high demand. 70 mph is closer to what is more typical even among baseball pitchers, and considerably slower throws would be typical of average people.
Now if you give him a stone instead of a baseball, the throw might not be quite as fast, but it will do more damage on impact.
I suppose so, but because of the way kinetic energy works you can't sacrifice much speed and not loss a lot of lethality. More on this in a bit.
Suppose a Giant trained in throwing object were given a lead ball, similar to a cannonball, and consider that damage he could inflict. what if the giant were simply using a sling appropriate to his size with lead bullets? wouldn't he then be comparable to some of the early cannons?
There are a ton of different assumptions being made here, and I'm not sure that any of them actually hold.
1) Can you train a giant? It's not clear that giants are as trainable as people. How long would you have to train a giant to achieve the same results? How willing would a giant be to endure training? What would the cost of feeding and placating a giant be over a period long enough to train them? What would the risks be?
2) Is the ability to throw stones transferable to any other sort of activity? It's not at all clear that the ability to throw stones is a learned activity, and is not an innate gifting that giants possess out of some magical affinity for stones. Particularly in the case of Stone Giants, it's not at all clear that they can throw as well or at all if the thing that they are throwing isn't a stone. The assumption that Giants are basically just big humans is not something that I think holds. It could be that giants throw stones well because they have some spiritual connection to stones that they wouldn't have to lead or steel balls.
3) Early cannon and trebuchet used stone for ammunition almost exclusively. If lead or iron balls were much more effective, they probably would have used them. Probably the increased effectiveness didn't justify the cost. So its probably pretty likely that the increased effectiveness of lead or iron ammunition wouldn't necessarily be worth the cost in a fantasy setting anyway.
4) Even if assume the use of a sling, it's not at all reasonable to assume that the giant can generate velocities much higher than those achieved by humans using the same mechanics. It's likely that the giant can throw a much larger weight, but it's equally likely that they cannot throw lighter weights faster or even throw their best weight as fast as human using the same mechanics. Giants are likely to be at least slightly less strong proportionate to their size as a human, and thus less likely to be able to accelerate weights up to the same velocities. I don't know exactly how fast a giant can throw, but it would be a mistake to assume that you could give them a 10lb cannon ball and they'd be able to throw it at 400mph or some such. Likely they can propel rocks at just 50 or 60mph. And that matters a bunch, because while kinetic energy is linear to mass, in most games you are going to assume that like the real world it's proportional to the square of the velocity. Even with a sling employed by some rare giantish slinger, your still dealing with a weapon that is more akin to what can be done with trebuchet and the like than something massively beyond it. Return fire from slingers, longbows, and torsion engines would likely be effective.
so we might see a development in fortresses similar to what happened in the real world where high stone walls were replaced by low wide earth and timber walls.
Err... you've skipped several faces of fortress development. I suggest you read about star forts if you want to see what the second phase of response to cannons looked like.
If the only way to defend against a creature, such as a ghost(who is unaffected by physical barriers), is through the use of magic then it wouldn't really affect the physical fortification.
I think attack by giants is going to be rare enough and insufficiently radical of a problem that it's going to move castle building to or beyond the star fort phase. You really need rifled guns and exploding shells to eliminate the star fort as an effective model. Late medieval castle builders could respond to giants in a number of ways - tame giants of their own, counter batteries making use of possessing the high ground to have superior range to attacking engines, sorties out to harass or kill attacking giants, or simply making their walls sufficiently stout that the attacking giant couldn't quickly reduce the walls. Remember, an army of attacking giants is going to require a lot of provender, and as such the besieging forces may run out of food and supplies before they can reduce the walls to rubble.
For the moment I'm taking for granted that all the necessary magical defenses are always used, but if a threat can be neutralized though physical defenses then that will often prove more economical and castles might look very different. I'm just not sure if there are some physical threats I'm missing out on.
Flying creatures dropping bombs is a much more absolute of a physical threat than giants, in that it's much harder to resist such attacks with anything less than flying creatures of your own.
