While I see your logic, and have gone down that road myself, the feat 'somatic weaponry' was created to avoid this.
-guess, I should have read the above post.
There was a great redux and some houserules that I use a lot in my games. It was over in Giant boards. can't post a link directly to it cause im too new, but if you google - 'giantitp.com:Drolyt Vow of poverty' fix it should come up.