Simplicity Rules
The house rule I use to give a game a more "novel" or "Cinematic" feel, or simply to allow the characters to truly be "heroic" in a great time of need, and cast a spell "beyond their normal ability" is relatively simple.
(Assuming the characters have no access to a Scroll...)
Spell Book Casters: Due to the form of magic they cast, they must research the spell they wish to cast and must cast it as a Ritual style spell. This does not lend itself to spontaneous need "power spells" but with time can allow them to do most anything.
Cost
* Research Time (up to the DM and the situation and resources available) but can range from an hour per spell level to days per spell level (or more).
* Casting Time: 10 minutes per spell level, Requires a successful Spellcraft check DC = 10+ Spell Level
* Spell Levels: Lose a number of spell levels equal to x1.5 (round down) the spell being cast, with lost spell levels always coming from highest spell level slots available first and working down to 1st level spell slots. If the caster does not have enough spell levels left, they are removed from the following day's available spell levels (preventing those slots from being prepared).
* Spell Requirements: Must still fulfill the material and XP requirements of the spell being cast.
* XP: to 5th level spells (50XP/spell level), over 5th level spells (100XP/spell level)
Example: A 10th level Wizard needs to cast a 9th level spell (once researched). This would take 1.5 hours to cast, will cost the required material/xp of the spell + an additional 900XP, and will burn 13 spell levels of prepared spells in the Wizard's mind starting with any 5th level spell prepared (for example the Wizard had 2 5th level spells as well as 2 4th levels still prepared, they would lose one 5th level spell and both 4th level spells).
Innate Casters: Since Innate Casters gain all of their spells from a internal source as natural magical ability, the costs for spontaneously casting spells beyond their ability is a bit different than Spell Book Casters.
Cost
* Instead of research time, the Sorcerer must spend time doing some serious soul searching, and contemplation. They must try to formulate in their mind the effect they wish to achieve and how they think this can be done. This process requires 30 minutes per spell level of the spell they wish to cast. In addition, once they have attuned themselves to the spell, this spell now becomes a “Known” spell for the Sorcerer. If the Sorcerer ever tries to cast this spell again while it is still “beyond their capability” they can forgo the 30 minutes per spell level preparation time. Once they attain a level were they would acquire a Known Spell of the spell level of this spell, this spell automatically fills that slot.
* Casting Time: 10 minutes per spell level, Requires a successful Spellcraft check DC = 10+ Spell Level
* Spell Levels: Lose a number of spell levels equal to x2 the spell being cast, with lost spell levels always coming from highest spell level slots available first and working down to 1st level spell slots. If the caster does not have enough spell levels left, they are removed from the following day's available spell levels (preventing those slots from being prepared).
* Spell Requirements: Must still fulfill the material and XP requirements of the spell being cast.
* XP: to 5th level spells (50XP/spell level), over 5th level spells (100XP/spell level)
Divine Casters: Since Divine Caster pray for their spells, they need only ask their divine source for the ability to cast their spells, simplifying the process by quite a bit.
Cost
* Pray Time: This is up to the DM but in dire circumstances this could be done as a Full Round action or take as long as a minute or two to formulate a prolonged prayer (and possibly a small sacrifice).
* Casting Time: As the desired Spell, Requires a successful Spellcraft check DC = 10+ Spell Level
* Spell Levels: Lose a number of spell levels equal to x1.5 (round down) the spell being cast, with lost spell levels always coming from highest spell level slots available first and working down to 1st level spell slots. If the caster does not have enough spell levels left, they are removed from the following day's available spell levels (preventing those slots from being prepared).
* Spell Requirements: Depending on the will of the “Divine Source,” may or may not need to still fulfill the material and XP requirements of the spell being cast.
* XP: to 5th level spells (50XP/spell level), over 5th level spells (100XP/spell level)