Been playing Elder Scrolls since Morrowind, and there's a pretty good list of the major points. First, I think you NEED the sandboxy feel. I love playing Skyrim and just stumbling on some random dungeon or quest line, and going from there. It's just fun times.
I disagree with people saying that this is a hard thing to do in actual play - it's not. You just need to have a few general dungeons kicking around within a few levels of your PCs, and adjust as you see fit. To play this sort of game, you need to be able to wing it... which happens to be my favourite part of RPGs.
Having characters be able to enchant their own gear, smith their own weapons, build their own houses, and progress in their preferred guilds are all good, "elder scrolls" ideas. And all of these are pretty easy to do in both 5th and Pathfinder, so that's a plus.
I've noticed that players in any Elder Scrolls game tend to be generalists with a focus on one area. So, everyone can cast healing magic, but some might be conjurers and others might just use a huge axe. Whether you want to do that, or just use the D&D class system is up to you. Personally, I'd just go with classes.
I love using alchemy in Skyrim. Stealing that system, and having players try to figure out what to combine with what to make potions would be pretty cool. It's like a campaign-length puzzle you can throw at them. And doing similar things with other parts of the game - enchanting and disenchanting items, finding metals to smith your own gear, and the like - could help your game feel "skyrimmy".
Also, I'd keep your individual quests short and sweet. In an Elder Scrolls game, the over-arching plot is something you visit occasionally, but you'll spend most of the time accomplishing short quests that are easy-in, easy-out.
I'd also pick a smaller setting, such as one province of Tamriel, and really just flesh it out. Personally, I'd go with one that hasn't been explored much yet. Somerset Isles? Elsewyr? Both would be cool. Apparently Elder Scrolls 6 is going to be in Orsinium, the Orc Homeland, so I'd maybe avoid that.
Give each place a theme to it. For example, in Skyrim, all of the holds have a different vibe to them. Riften is a wooden Venice that's home to con artists and thieves. Winterhold is built on an icy cliff and has a bunch of decaying buildings and a frozen castle full of mages. There's the city of Solitude, a city of "civilization" in the north (it reminds me of Baldur's Gate). And then there's Morthal and Markarth, which I always get confused - one's a horror-vibe swamptown plagued by ghosts and vampires (with a young heir who seems to have epilepsy!), while the other is busily involved in an uprising from the local natives. If you can provide one "hook" to each settlement, you're well on your way to an Elder Scrolls game.
I mean, every time I play one, I wind up picking one place that I just tend to gravitate to. In Skyrim, I always try to hangout in Rorikstead whenever I get the chance, and Dragonsbridge is awesome.
I'd also make sure that you fill up your areas with awesome wandering monster lists, and have those lists trigger some special encounters that you prep ahead of time.
Finally, if I were you, I'd go on the wikis for Elder Scrolls, and liberally steal a bunch of quest lines from them, and convert as you see fit. That'd be an easy way to really capture the game's feel. Make sure you steal the pirate ship/inn encounter from Oblivion! I love that questline.