I'm not familiar with the adventure you're playing so I can't speak to it specifically, but traditionally the issues you're dealing with are dealt with two ways:
1) low level (especially 1st level) characters should only be adventuring in chaotic and disorganized environments, where the bad guys don't have effective reaction strategies (or are so badly organized or mutually antagonistic that they won't implement them effectively even if they do have them) and the dungeon should have lots of empty space and not be regularly patrolled; between these two factors a party that plays intelligently (employing stealth and effective scouting, not getting caught without an escape route or leaving obvious trails for the bad guys to follow, etc.) should be able to retreat, rest, and come back later. In other words, for low level parties the DM needs to "stack the deck" in creating a forgiving environment, one which is uniquely situated to allow parties to make multiple forays without the place turning into a fortress. That said, however, even a group of completely disorganized bad guys are likely to have reacted in at least some way to the party's initial foray, so the second and subsequent forays will (or at least should) be tougher than the first -- less chance of achieving surprise, reinforcements will come sooner, more guards in key areas, etc. -- which leads us to point
2) the party should use their resources effectively so as to minimize risk and exposure -- if you've only got sufficient resources to last through four encounters, you want to get as much accomplished with those resources as possible, meaning if you're exploring a 60-room dungeon you don't want to just fight the first four monsters you meet, retreat to rest, and come back 15 times, but rather you want be strategic about it, to avoid or circumvent as many non-essential encounters as possible to maintain your resources for those encounters that are essential, allowing you to complete the dungeon in as few expeditions as possible -- a 60-room dungeon is likely well beyond the means of even a very careful and clever party to beat in a single expedition, but might be possible in 2 or 3, whereas a more careless party might take a half dozen or more (remembering, as above, that the dungeon is likely to get a little tougher with each new foray as the inhabitants react to and learn from the party's actions -- take too long, allow the bad guys too much time to react, and eventually the dungeon will become impossible (I've had this situation occur a couple times in my games)). So, a clever party will always be looking to run away from, sneak past, trick, negotiate with, or bribe minor/non-essential encounters so they can get past them without having to use up their crucial combat and magic resources; wandering monsters become the ultimate bane who should always be run away from and never fought, because they use up resources without accomplishing anything (you're no closer to your goal and have now used up 1/4 of your resources needlessly).
If you have a 1st edition DMG (and if you don't you probably should, for reasons just like this) there's a great essay (pp. 104-105) about Monster Organization and how monsters of various intelligence, alignment, and organization levels are likely to react to repeated forays by adventurers which should give a good idea of how to handle these sorts of situations (and emphasize the fact that, given sufficient time and foreknowledge, even a group of statistically weak opponents can challenge a much tougher party if they're of sufficient intelligence and organization).
This also ties into your other point about "when will intelligent monsters just give up?" That's a very good question which should always be kept in mind. The answer, of course, depends on the monsters -- their strength, their discipline (and alignment), their motivation, etc. A group of chaotic bandits who are camped out in or near the dungeon because they're planning to raid it themselves probably won't stick around in the face of a threatening party; a group of weak lawful monsters (like goblins or kobolds) will probably have good morale as long as the leader-types are present, but will fold quickly once those figures are defeated or neutralized; a group defending a special lair (a holy site, ancestral burial ground, etc.) will stick around longer than a group who is just lodging in a random set of caves/ruins. Remember also that a group of monsters that does decide to pack up and leave the dungeon will of course take all of their best and most valuable treasure with them (and will attempt to hide what they can't take with them), and may leave booby-traps behind out of spite (or with an eye towards a possible eventual return). And just because one set of monsters has left doesn't mean another set won't take their place (especially if the location being abandoned is "prime real estate" for monsters -- easily defensible, near a water source, etc.).
