I blame hierarchical command structures. Every leader who has ever authorised an atroctiy has been part of one, every soldier who has ever participated has been a part of one, and every citizen who ever ignored an abuse was part of one.
BAN HIERARCHICAL COMMAND STRUCTURES!!
Read some modern organisation and management literature and you might be surprised about the core of truth in your broadly generalized statement. 

Well, to be honest, I wasn't entirely picking a subject at random, there were times I would have happily fed my manager into a grinder, and that's violence generated by the command structure

The thing about Hierarchical command structures is that, like religion, they have both also had massive positive effects on the world we live in and are, for the main part, harmless. It's a problem that is encountered by Researchers a lot, the fact that you have to differentiate between 'A contains B' and 'B is a part of A'. Or, for a more interesting perspective, if you destroy A, would B cease to happen and vice versa? Since the answer, to me, seems to be 'No', I think that whilst we should be concerned about the Meme-transmitting ability of religion, and the kind of memes it is getting into the habit of transmitting, going on a seek and destroy mission is really picking the wrong target just as blaming Earthquakes on Gays is.