I'd like somewhere between this and how 13th Age does it.
You have 4 attunement slots. However, attuning an item of greater rarity costs extra slots. This keeps the concept true while still limiting items.
First tier (1-4) common items take 1 slot (of which I don't think there are any), others take more.
Second tier (5-10), uncommon items take 1 slot, others take more.
Third tier (11-16), rare and uncommon items take 1 slot, others take more.
Fourth tier (17-20), very rare and lesser items take 1 slot, others take more.
Each step above what you can wield takes an extra slot. Common, uncommon, rare, very rare, legendary.
Yes, this particular setup means that legendary items always take multiple slots. At 17th it takes two of your attunement slots leaving you with two left. Luckily you started up one with this plan.
So a 1-4 level character could have two uncommon items attuned. Not bad. Or even one rare or very rare.
At 5th to 10th you're sort of around where you are now. You could have four uncommon items, but a rare and two uncommon works. Very rare and common or two rare also work.
11th to 16th you doing slightly than now in almost all realistic cases.
At the last tier of play, for all intents you have a full other slot over what you have now, unless you are blessed enough to have a legendary item, in which case just be happy.