I always find this kind of reply interesting. Why would the other players object to a potential adventure simply because it's tied to one of the PC's background? Yet, if the GM introduced the idea of a mysterious island, they'd all likely be on board.
The objection seems to be the connection to one character. Why is that?
Because it seems like your catering to one person more then the rest for one thing. And because that characters backstory has nothing to do with anyone else. So why should the other characters risk their lives for it and put their other goals on hold to chase after one guys past?
When your playing a sandbox campaign theres usually lots of things the party can choose to go do at any point in time. So why put the goals of the whole group behind the goals of one person?
If that player can sell the rest of the party on chasing down their personal demons then by all means sure. But I'm not making that sales pitch for them and I'm certainly not forcing anyone else to do it.