Yes, brutal, eh? He only takes damage if the defender uses Weaponry to defend against the Unarmed attack. But yes, good reason to specialize
Of course one can adjust to taste
I guess I'll have to give it a chance. But it seems too much to me. I favor eliminating the "attacker takes damage if the defender uses Weaponry to defend against the Unarmed attack" part. Possibly one way to rationialize this: If the foe takes an attack of opportunity, he cannot use his Weaponry ADP to defend (because he has used his weapon to attack, not defend).