Agree with [MENTION=6803538]HermanTheWize[/MENTION] that 20 Dex should be your next priority. (Though if you have an odd score, a half-feat to boost it and give you something is a useful touch.)
The Precision Attack battlemaster maneuver should help a lot with delivering your Sharpshooter attacks, but a better chance to hit will always help. Plus more hits and more damage regardless if you are using the -5/+10.
I'd say stay pure fighter - it's a strong chassis with many ASIs, and also will grant you Extra Attack (2) and eventually (3). That's assuming your campaign is expected to go past 11th - if it will end before then you may want to reconsider. But 5th is needed for Extra Attack, 6th is an ASI for your Dex, 7th is another Superiority die and Know Your Enemy, and 8th is yet another ASI - at that point staying for 11th is pretty easy.
If you do multiclass, I'd stay until at least 8th. Also, since you won't be getting the advanced version of extra attack, you might want to pick up Crossbow Expert for a bonus action attack and switch the to hand crossbow. It works fine with SS, and SS removes disadvantage at ranged so the hand crossbow is good up to 120'. (And if it's further, just use your bow.)
Not being able to use magic does remove some multiclassing, be it to Ranger for one of the subclasses with good features for an archer, or a dip for Hex, Shield, or other non-casting-ability score buff or utility.
In addition to the feats others have suggested, if your DM allows XGtE take a look at Fey Teleportation. It's a free get-out-of-melee card once per short rest, and has plenty of creative uses in and out of combat. Don't know if that hits into your "can't use magic" or if more High Elf features are okay like the cantrip is okay.
