Yes, I think that is the issue. Birthright just do not sit well on top of D&D due to the existing associations and somewhat squeaky connections. It is presented as D&D with a domain subsystem on top, while I think it is really better suited as a domain game with a subsystem of personal attributes facilitating diving into single character roleplay.
And I think the later might have a die hard following. Granted, crusader kings might not be as popular as skyrim - but it is not exactly a fringe game either.
But if people are approaching it as another monster smashing treasure hunting setting, I think they are bound to be disappointed. Hardly anything in Birthright really support that kind of play. The other approach I have seen to Birthright domain play is as a board game, which I again think completely misses the point. As a board game the system is a real stinker.
So I think Birthright only really works well using the style you describe. But it do not appear to make a good job of educating people into this style, and hence people have had poor experiences with it.