Granted you could read the feat that way, but as...a whole bunch of people have said, it would diminish the feat greatly.
As for your problems with the feat, I'd say start with 1-4 encounters with these creatures acting dumb this way, as they have no idea what the characters are capable of. After this, perhaps the characters will build a reputation and the goblyns will send small attack forces of their powerful champions, or assassins to snipe the characters from a ways away, while smaller melee skirmishers lie in wait if the PCs decide to try and kill the snipers.
To counter the bane weapons, perhaps have the goblyns hire giants or train attack dogs so the characters don't get this huge bonus any time they fight.
But if you still think the cramped quarters fighting with wimpy goblyns aginst the built up PC force is the way to go, then I suggest multiplying the number of creatures by 4 for CR 7 making it CR 11, or by 8 or 16 for the CR 4 encounters to make them challenging (do I have my CR rules right there?)
Just mix up the enemies, mix up the tactics, and give them strange abilities...keep the PCs guessing, and you'll challenge their full range of abilities more.