Is it OK to tell a 12yo to defend themselves with greater force than they were attacked with?

Yep, I agree. If someone is hitting on you, then you hit their as* back. Nobody should take physical abuse from anyone and the school needs to be informed that your nephew is being bullied. Because if he does hit the kid, then that kid could play innocent & blame everything on your nephew.