Depends on the weakness. If it's a commonly known and easily acquired weakness such as a Skeleton's damage vulnerability to bludgeoning weapons it doesn't really make much difference. It's easy enough for a fighter to just switch to a mace or maul and whack away.
If it's a weakness that's hard to exploit like, say, a monster that regenerates unless struck by enchanted jade and is immune to most everything else that's a lot trickier. Either the PCs know the trick to dealing with the thing or they don't.
More likely, it's something like being Resistant or Immune to nonmagical bludgeoning/piercing/slashing damage. A low-level party with few or no magical weapons could only use spells or "out of the box" thinking to deal with such a menace. A mid to high level party is probably fully equipped with enchanted arms & armour and could deal with one relatively easy.
Take, say a
Gargoyle. In 5E those are resistant to nonmagical attacks that aren't adamantine, so a typical bunch of 1st level noobs would be doing half damage against it with their weapons and it'd be hell to deal with.
At least it isn't Immune to all nonmagical weapons like the AD&D version of the Gargoyle, which requires +1 or better weapon to hit. A party without magic would be forced to try to drown the thing in a bathtub or forcefeed it poison or something (although the latter wouldn't work in 5E which gives gargoyles immunity to petrification and poison).
Actually, a Gargoyle is a decent example of a 5E monster that becomes a bit of a scrub down the line once a typical party will usually be magic-wielding.
By comparison, a Carrion Crawler* is also Challenge 2 in 5E and arguably keeps its threat value better since it has a paralysing poison so there's always the change its target might fluff their saving throw.
*Unfortunately, crawlers aren't SRD so I can't include a link to its stats.
A pack of CR 1
Ghouls might theoretically offer a similar threat to half their number of Carrion Crawlers judging by their CR, but ghouls are a lot easier to blow away having only 22 hp to a Crawler's 51 hp plus a decent level cleric can obliterate them with Turn Undead, so could also be considered another creature that loses its impact at higher levels, although not as much as a Gargoyle since there's always the chance of flubbing that DC 10 saving throw versus paralysis.