Well, the more immediate lord might have cared because it meant less income and, if it went on, made him look like he couldn't protect his lands.
I think the one reason you didn't see so much terrorism in the old days is that people back then were much more ruthless and brutal in dealing with...