When I see questions like this, I always think of my sig. Frank, flavour text can justify *anything*. Your statement is flavour.
What is the Fun Factor? Do your PCs like it when the odds are stacked against them? (This is not a sarcastic question; many PCs *do* like that challenge.) Then limit the spell similar to your description. If not, let them cast it.
What is the Balance? Here, imo, the situation is clear. Feather Fall is one of those "lifesaver, in very very very limited situations" spells. Most of those types of spells usually go on scrolls; but Feather Fall is unique (in the SRD) in that the situation it is most needed for makes a scroll useless. Thus, a wizard must prepare it, or a sorceror pick it as one of his very limited spells. In either case, the wizard or sorceror could go days, even weeks, of campaigning without using it even once. In my opinion, if the wizard/sorceror is using the spell, and the situation comes up where it *could* be useful, but the rules are ambiguous--then let it be useful. Otherwise, Feather Fall becomes an all-but useless spell except in very very specific situations ("We must climb to the top of Mount Killemall to get the MacGuffin to defeat the BBEG! Wizard, prep a few Feather Falls." "Why? I'll just Teleport us up." "...Why did you get Feather Fall anyways?" "Dunno. Never used it." : ) and the caster just memorizes another Magic Missile.
What are the Mechanics? By RAW it's unclear, as others have demonstrated. The flavour can be worded any which way you want, from letting the caster use it any time he's alert, to never letting him cast it until his action comes up in a combat round (and since combat doesn't happen with a pit trap, it can only be used in combat situations...)
In my campaigns, I would let the caster cast the spell in almost any appropriate situation, without creating additional limiting house rules. As another poster said, that's what Feather Fall was specifically *designed* to do!