I would say that the "neither worn nor carried" language strongly implies that items which are worn or carried are not affected.
Perhaps, if that was all that was said.
But the text on p 87, which says that "Characters can also damage objects with their weapons and spells" and that objects "can be affected by physical and magical attacks much like creatures can" not only strongly implies, but actively asserts, the contrary.
In light of the rest of the discussion on p 87 (about the GM deciding an object's hp and resistances), the point of the text about igniting flammable objects seems to be to take the matter out of the realm of discretionary adjudication and to stipulate an outcome.
This is also supported by the definition of fire damage - it hurts because it involves flame, like a dragon's breath - and that definition is in turn reinforced by the descriptions of burning oil and the like, which we know are flaming things and which are said to deal fire damage.
Suppose that a Burning Hands spell deals 10 hp of damage to a Cultist (9 hp) who is trying to run from the cult hide-out carrying the scroll with the ritual for summoning some otherworldly being. The cultist drops below 0 hp. Page 98 of the SRD states:
Most GMs have a monster die the instant it drops to 0 hit points, rather than having it fall unconscious and make death saving throws. Mighty villains and special nonplayer characters are common exceptions
This cultist is, in the view of the GM, neither a mighty villain nor a special NPC and so the GM narrates the cultist as dead, with skin burned and hair charred.
Is the GM house ruling if s/he also describes the cultists clothes as burned and blackened? Personally, I don't see how the cultist could be burned to death yet his/her clothes remain unaffected.
Is the GM house ruling if s/he described the scroll as being burned? My response is the same - how can you wreathe a cultist in flames sufficient to kill him or her, yet there be
no chance that the paper s/he is carrying be set alight?
That's not to say that the paper is obviously destroyed, either. But surely there is some chance of that?
In a situation in which the NPC is damaged (ie loses hp) but does not die, further complexities in adjudication arise because there is ambiguity in what hit point damage consists in, what a DEX saving throw amounts to, what the AoE is (it is a full cone, as the rules text implies, or a "thin sheet" as the flavour text asserts?), etc.
On the Gygaxian account of hit points, it is arguable that the scroll can't be damage for the same sorts of reasons as there being no need for a hit location system (AD&D DMG, p 61):
[H]it points are not actually a measure of physical damage, by and large, as far as characters (and some other creatures as well) are concerned. Therefore, the location of hits and the type of domage caused are not germane to them. . . . Damage scored to characters . . . is actually not substantially physical - a mere nick or scratch until the lost handful of hit points are considered - it is a matter of wearing away the endurance, the luck, the magical protections.
But not everyone uses Gygaxian hit points; and the 5e default is that physical damage begins at 50% loss, not just the last handful (Basic PDF, sidebar on p 75). Furthermore, even with Gygaxian hit points a "nick or scratch" from fire damage could be sufficient to set a piece of paper alight!
I think there could be different ways of handling this. Rolling a save for the object appears to be ruled out (because objects automatically fail their DEX saves, and are immune to effects that require a CON save - and what other sort of save would be applicable here?), but the GM might call for an INT/CHA check to target the spell so as to avoid (or, if desired, to ignite) the scroll - because INT/CHA is the stat that affects casting for wizards or sorcerers (as appropriate to the PC's class). Or the GM could deem that the paper has d20, or 2d10 hit points, which becomes functionally equivalent to a saving throw vs the spell's damage. Or the GM could say there is a N% chance the paper is caught within the sheet of flame - say, 30% if the NPC is damaged but not killed by the spell, and 70% if the NPC is dead from being wreathed in flames.
I don't think the rules dictate any of these approaches - though the hit point one seems most canonical, given the text on p 87 of the SRD. But I don't think that a GM trying in these sorts of ways to make sense of the fiction in light of the rules is house ruling. S/he is just adjudicating the game.
And all the moreso when s/he allows a lit torch to ignite another torch (say, in circumstances where the PCs have lost all their tinderboxes but don't want to be left without light).