Is it that a spectral weapon cannot be attacked, or that attacks against one won’t do anything? Certainly, an arrow could be shot at one.
An arrow is an object, but a spiritual weapon is not one, it's just a spell effect, so theoretically and according to the rules, although you can shoot an arrow at one, it cannot be attacked (in the technical sense of the game) and therefore cannot be damaged.
You might, however - and this is one of the beauties of 5e, although some consider it a curse
- decide that spiritual weapon, as it creates a weapon, and weapons are usually objects, creates an object shaped like a weapon. My take is that is not what is intended here, since spells only do what they say they are doing, and spells which create objects specifically say that (see for examnple fabricate). It also has the drawback (which is one more reason not to interpret the spell that way) of leaving you to define the characteristics of the objects. YOu might use those of the weapon created, but first these are not precise in the books, and it also starts to be discriminating towards the divinity, some spiritual weapons becoming more powerful and resistant than others.
Backing up a little, would a wall of force block the movement of a spiritual weapon? I can accept that the visible manifestation is akin to the dot from a laser pointer, arriving from unseen dimensions. Then, a block attempt would have no more effect than a drawn line would have for blocking the movement of the dot from a light pen. Or, a spiritual weapon might really be there, in a fashion similar to arcane hand. But, being spectral, most interactions wouldn’t do anything to it.
Wall of Force is a tricky spell. It blocks everything physical, so it provides total cover, and you therefore cannot cast spiritual weapon behind it, since you need a line between you and your spell effect. However, as is the case for example with fireball, if you cast it somewhat to the side, the effects can extend behind the wall, since the restriction is at casting time only. You could cast spiritual weapon to the side and then move it behind the wall for example. But you could not cast it inside the sphere.
Finally, it becomes more an interpretation, but both wall of force and spiritual weapon are "force" and both physically affect the world, the wall by blocking things and the weapon by hitting things physically. It stands to reason that the wall would block the weapon as being physical, but since the weapon is also "spectral", discussions can be had forever as to whether it refers to its appearance, or it's materiality, although you could also consider that "spectral" refers to the ethereal plane (like spectres or ghosts) and is therefore blocked anyway by the wall. 5e is not that precise and these are rulings for your table.
Regardless, that doesn’t mean an opponent (or character) would know that an attack would be ineffective.
It's a floating spectral weapon, it's not illogical that people would think that their attacks would be ineffectual. Also, it does not move or dodge except as commanded, so I assume that just waving something through it without even attacking (consider it "a flourish" that you can use for free during your turn) would reveal it as spectral and immune to your attacks. Even animals would see this quickly, if you want to use some rationalisation here.
Xanathar's has a thing about choosing invalid spell targets (pp 85-86) that IMO you can extend to targeting perceptible phenomena with attacks. That's how I'd rule such things if they came up in a game.
I have not read that ("extend to targeting perceptible phenomena with attacks") in Xanathar at all. The whole section reads: "A spell specifies what a caster can target with it: any type of creature, a creature of a certain type (humanoid or beast, for instance), an object, an area, the caster, or something else. But what happens if a spell targets something that isn’t a valid target? For example,
someone might cast charm person on a creature believed to be a humanoid, not knowing that the target is in fact a vampire. If this issue comes up, handle it using the following rule. If you cast a spell on someone or something that can’t be affected by the spell, nothing happens to that target, but if you used a spell slot to cast the spell, the slot is still expended. If the spell normally has no effect on a target that succeeds on a saving throw, the invalid target appears to have succeeded on its saving throw, even though it didn’t attempt one (giving no hint that the creature is in fact an invalid target). Otherwise, you perceive that the spell did nothing to the target."
And that's all. Maybe you refer to something else ?