Honestly I think this does sound like a good idea - in fact it's something I've allowed in my own campaigns. As long as you're a caster, you can use any scroll but you have to make the related check. I wouldn't extend this to non-casters though, because while the spell and the magic are on the scroll the character still needs some way to activate it. The way I see it is, even if a spell is pre-made, you still have to have the capability of actually casting a spell.
Another possibility you might toy with is allowing a spell scroll to be used by anyone with the appropriate type of magic. For example, a druid and a ranger both use nature based magic, cleric and paladin use divine magic, wizards, sorcerers, and warlocks use arcane magic (not 100% sure where a bard would fit in, because they cast arcane magic but they can also steal spells from other spell lists). Then, if they encounter a scroll of the right type of magic they can use it so long as they meet the DC for the spell level and it's of the appropriate magical type. If that makes sense.
Either way, I find it's generally best to let people use spell scrolls they wouldn't normally be able to in smaller groups or groups that only have one or two casters. In groups like that they're very limited in the spells they can use by default, and unless you're deliberately only giving them scrolls from their spell list they won't be able to use most of the ones they find.