“I can’t bear to hear a human being spoken of with contempt just because of his group identification-even by other human beings. It’s these respectable people here who create those hooligans out there.”
The quote is from Dors from “Prelude to foundation” by Isaac Asimov.
“These respectable people here” refers to the snobbish class of people who made these comments. The “hooligans out there” refers to the marginalized part of the community who has developed violent behaviour.