Counterpoint: The spectral weapon is clearly a spell effect -- not a creature -- but who says you can't fight back?
For example, can the weapon move through solid objects? The rules don't say that it can't, but most DMs would be kinda surprised if a player directed the weapon to go straight through a wall to get to someone on the other side. If the weapon can't move through solid objects, what happens if a solid object suddenly fills the space the weapon is in (maybe the ceiling collapses or something)? Does the weapon blink out, or get shunted aside? The spell text is very silent on these issues. Let's say that you decide the weapon gets shunted aside: well, why couldn't I shunt the weapon aside with my shield, pushing it into a different position, or even pinning it to the ground? There are other movable force effects in the game (Tenser's, Otiluke's). Let's say you decide that the weapon blinks out; why can't I smash my hammer into the weapon, such that it goes through the weapon, blinking it out? Suppose you rule that the weapon can move through and exist inside objects just fine; in that case, when I fight it, it would be like fighting a "damaging illusion" where my attacks go through it but it can still hurt me. I think most creatures would waste at most 1 round with that thing before running away or seeking a different target.
I'm not sure how I would rule on any of this stuff, but I think ruling that you can push/pull/grapple the weapon is at least as reasonable as saying that the weapon is invisible except when attacking, or that it has invisible strings back to the caster. I know in the past I've told players "You can't attack the weapon, it's a spell effect," but if someone had a clever idea (e.g. trapping it in a cage) I'd probably let it roll.
For example, can the weapon move through solid objects? The rules don't say that it can't, but most DMs would be kinda surprised if a player directed the weapon to go straight through a wall to get to someone on the other side. If the weapon can't move through solid objects, what happens if a solid object suddenly fills the space the weapon is in (maybe the ceiling collapses or something)? Does the weapon blink out, or get shunted aside? The spell text is very silent on these issues. Let's say that you decide the weapon gets shunted aside: well, why couldn't I shunt the weapon aside with my shield, pushing it into a different position, or even pinning it to the ground? There are other movable force effects in the game (Tenser's, Otiluke's). Let's say you decide that the weapon blinks out; why can't I smash my hammer into the weapon, such that it goes through the weapon, blinking it out? Suppose you rule that the weapon can move through and exist inside objects just fine; in that case, when I fight it, it would be like fighting a "damaging illusion" where my attacks go through it but it can still hurt me. I think most creatures would waste at most 1 round with that thing before running away or seeking a different target.
I'm not sure how I would rule on any of this stuff, but I think ruling that you can push/pull/grapple the weapon is at least as reasonable as saying that the weapon is invisible except when attacking, or that it has invisible strings back to the caster. I know in the past I've told players "You can't attack the weapon, it's a spell effect," but if someone had a clever idea (e.g. trapping it in a cage) I'd probably let it roll.