Technically, if you are using weapon attacks, then your 'free' object interaction is being used on the weapon you used. I can see plenty of DMs deciding that on the spot. I've seen plenty DMs rule that if you spent your turn opening a door you couldnt attack because your weapon had to be...