Here's how I see it:
I went to a good school, because I was one of the "cool" kids. There was always a crew of people who were unpopular. If someone made a demeaning joke about someone else, sometimes I'd laugh. I wouldn't hang out with them even if they specifically asked me, despite being friends in class. I still don't reply to 90% of the hellos one guy says on steam chat because he's gay. I don't consider myself a bully because I never go out and try to put someone down. But it isn't the one guy who makes a joke at your expense, the one girl who calls you by a degrading nickname, or the one time you get beaten up because that dickhead is actually a blackbelt, no it's the bystanders who make your life a misery. Everyone looks down on you and views you as an acceptable target, and those people who don't think about you are worse than those who actively degrade you.
It is so, so easy to ignore someone, laugh at a joke or avoid someone, yet multiply it by many people and suddenly someone feels, quite rightly, that the world is against them.