My stab at the 'why' part: -- I just checked my old games, and while Lycanthropes lose their immunity to non-silvered, non-magical weapons while in human form in the basic/classic line, but AD&D does not follow suit. 5e was designed, at least in part, to recapture lapsed TSR-era gamers who had drifted away in 3e or 4e (or earlier), since they had no notion that 5e would be such a hit with new gamers (and their ambitions for the game were more than just 'keep they buyers we have'). Thus, there are lots of little nods, winks, and throwbacks to the way things once were. Not sure if emulating AD&D rather than (for example) B/X has any meaning, or it was just a 'had to chose one way or the other' situation.
Beyond that, near invulnerability (but only when in a specific mode/stance/configuration/form) works better for narratives than games. It's great in comic books (Hulk/Bruce Banner, who in the comics usually can be killed while in human form), tv shows (Dinobots in the original Transformers), or the books (there's a scene in Clive Barker's Cabal where the quasi-undead/lycanthrope protagonist is about to face the main antagonist 'as a man,' and then has a 'what on Earth am I doing?' moment before turning into his neigh-invulnerable form and greasing him). It allows the otherwise scene-ruining character to be taken out as needed by the plot, but the author will always give them enough warning to transform when necessary. In a game, it just incentivizes characters to stay in the invulnerable form at all times 'just in case.' Mind you, the game/DM could make that hard (similar to druid forms and spellcasting) and that be part of the fun, but I suspect that the devs expected less of that and more of the old 'novice DM insists that this town does not allow adventurers to wander around with weapons and armor when and only when they they want to spring a PCs-captured-and-in-jail-cell scenario, and everyone hates it' situation.
I would have preferred that not be there. That said, it does mean that it is the DM deciding to include this portion of the adventure, and a DM that does so had best be prepared for the fallout. It's a step less 'communicated norm' than having werecreature as a PC race option for building new characters (although DMs should be comfortable vetoing both events in an adventure and unbalancing character-creation options). Given that plenty of newer DMs are the ones who run modules, I would prefer that options that can easily cause problems were presented and suggested. Let the PCs decide to experiment with subjecting themselves to lycanthopy on their own. My own recollection of my early gaming experience suggests it won't take too long for most groups.
Fundamentally, I think D&D (from a very early stage) mishandled lycanthropy*, vampirism, and being a lich. They are 3 qualities that the PCs clearly can acquire. They are acquirable through relatively clear means (the first two by capturing an existing one and saying 'make me a vampire/werewolf'), so it is harder (short of just saying 'I don't want to run that') for the DM to keep them from it (compared to say, membership in a secret society or the like). Yet the power level of the specific implementation of the base concepts are so far towards the upper end (vampires get a huge array of powers from the folklore, werewolves are low-tier invulnerable, liches are low-tier immortal). It could have been that vampires gave a bite attack and one other vampirish ability (the Dhampir lineage option is a good example), Werewolves could resemble the Shifter race, and Liches could be like 3e Necropolitans. IMO that would have worked much better for things PCs could (relatively) easily get, and reserve the higher-power abilities for things easier to gatekeep (an 'Elder Vampire' MM entry would be easy to say "sorry, your character will get these powers in a few centuries if they stay as a vampire, but we won't be playing that any time soon").
*the curse gained from a bite, being a native-born one can work however they need.