when the rope trick space tries to contain the bag's space, the bag responds by trying to contain the rope trick and everything in it. This results in one of the following conditions:
a. the rope trick wins; the bag is constrained to fit within and behave normally. no problem.
b. the bag wins. the characters and their possessions go inside the bag. if they fit, this is just like being inside the rope trick except that outsiders see the bag lying on the ground, and can interact with it. if they are too big to fit, it is because they occupy too much space-time for the bag to contain. since their space is fairly inelastic*, they need to occupy less time in order to fit. the end result is that the rope trick ends within a round or two and all the characters are dumped onto the ground.
c. the two effects reject each other. the bag simply cannot be made to enter the rope trick.
d. the bag and the rope trick space breed, spawning tiny pocket dimensions that grow by eating pieces of reality. major quest to get rid of them before they eat the world.
*a cruel gm could decide instead that the space taken up by the characters can in fact be made to fit inside the bag by shrinking, warping or liquifying them as needed.