By Julia Angwin, ProPublica, and Hannes Grassegger, special to
In the wake of a terrorist attack in London earlier this month, a U.S. congressman wrote a Facebook post in which he called for the slaughter of "radicalized" Muslims. "Hunt them, identify them, and kill them," declared U.S. Rep. Clay Higgins, a Louisiana Republican. "Kill them all. For the sake of all that is good and righteous. Kill them all."
Higgins' plea for violent revenge went untouched by Facebook workers who scour the social network deleting offensive speech.
But a May posting on Facebook by Boston poet and Black Lives Matter activist Didi Delgado drew a different response.
"All white people are racist. Start from this reference point, or you've already failed," Delgado wrote. The post was removed and her Facebook account was disabled for seven days.
A trove of internal documents reviewed by ProPublica sheds new light on the secret guidelines that Facebook's censors use to distinguish between hate speech and legitimate political expression. The documents reveal the rationale behind seemingly inconsistent decisions. For instance, Higgins' incitement to violence passed muster because it targeted a specific sub-group of Muslims — those that are "radicalized" — while Delgado's post was deleted for attacking whites in general.
Over the past decade, the company has developed hundreds of rules, drawing elaborate distinctions between what should and shouldn't be allowed, in an effort to make the site a safe place for its nearly 2 billion users. The issue of how Facebook monitors this content has become increasingly prominent in recent months, with the rise of "fake news" — fabricated stories that circulated on Facebook like "Pope Francis Shocks the World, Endorses Donald Trump For President, Releases Statement" — and growing concern that terrorists are using social media for recruitment.
While Facebook was credited during the 2010-2011 "Arab Spring" with facilitating uprisings against authoritarian regimes, the documents suggest that, at least in some instances, the company's hate-speech rules tend to favor elites and governments over grassroots activists and racial minorities. In so doing, they serve the business interests of the global company, which relies on national governments not to block its service to their citizens.
One Facebook rule, which is cited in the documents but that the company said is no longer in effect, banned posts that praise the use of "violence to resist
One document trains content reviewers on how to apply the company's global hate speech algorithm. The slide identifies three groups: female drivers, black
The reason is that Facebook deletes curses, slurs, calls for violence and several other types of attacks only when they are directed at "protected categories" — based on race, sex, gender identity, religious affiliation, national origin, ethnicity, sexual orientation and serious disability/disease. It gives users broader latitude when they write about "subsets" of protected categories. White men are considered a group because both traits are protected, while female drivers and black children, like radicalized Muslims, are
Behind this seemingly arcane distinction lies a broader philosophy. Unlike American law, which permits preferences such as affirmative action for racial minorities and women for the sake of diversity or redressing discrimination, Facebook's algorithm is designed to defend all races and genders equally
"Sadly," the rules are "incorporating this color-blindness idea which is not in the spirit of why we have equal protection," said Danielle Citron, a law professor and expert on information privacy at the University of Maryland. This approach, she added, will "protect the people who least need it and take it away from those who really need it."
But Facebook says its goal is different — to apply consistent standards worldwide. "The policies do not always lead to perfect outcomes," said Monika Bickert, head of global policy management at Facebook. "That is the reality of having policies that apply to a global community where people around the world are going to have very different ideas about what is OK to share."
Facebook's rules constitute a legal world of their own. They stand in sharp contrast to the United States' First Amendment protections of free speech, which courts have interpreted to allow exactly the sort of speech and writing censored by the company's hate speech algorithm. But they also differ — for example, in permitting postings that deny the Holocaust — from more restrictive European standards.
The company has long had programs to remove obviously offensive material like child pornography from its stream of images and commentary. Recent articles in the Guardian and Süddeutsche Zeitung have detailed the difficult choices that Facebook faces regarding whether to delete posts containing graphic violence, child abuse, revenge
The challenge of policing political expression is even more complex. The documents reviewed by ProPublica indicate, for example, that Donald Trump's posts about his campaign proposal to ban Muslim immigration to the United States violated the company's written policies against "calls for exclusion" of a protected group. As The Wall Street Journal reported last year, Facebook exempted Trump's statements from its policies at the order of Mark Zuckerberg, the company's founder and chief executive.
The company recently pledged to nearly double its army of censors to 7,500, up from 4,500, in response to criticism of a video posting of a murder. Their work amounts to what may well be the most far-reaching global censorship operation in history. It is also the least accountable: Facebook does not publish the rules it uses to determine what content to allow and what to delete.
Users whose posts are removed are not usually told what rule they have broken, and they cannot generally appeal Facebook's decision. Appeals are currently only available to people whose profile, group or page is removed.
The company has begun exploring adding an appeals process for people who have individual pieces of content deleted, according to
The law freed up online publishers to host online forums without having to legally vet each piece of content before posting it, the way that a news outlet would evaluate an article before publishing it. But early tech companies soon realized that they still needed to supervise their chat rooms to prevent bullying and abuse that could drive away users.
America Online convinced thousands of volunteers to police its chat rooms in exchange for free access to its service. But as more of the world connected to the internet, the job of policing became more difficult and companies started hiring workers to focus on it exclusively. Thus the job of content moderator — now often called content reviewer — was born.
In 2004, attorney Nicole Wong joined Google and persuaded the company to hire its first-ever team of reviewers, who responded to complaints and reported to the legal department. Google needed "a rational set of policies and people who were trained to handle requests," for its online forum called Groups, she said.
Google's purchase of YouTube in 2006 made deciding what content was appropriate even more urgent. "Because it was visual, it was universal," Wong said.
While Google wanted to be as permissive as possible, she said, it soon had to contend with controversies such as a video mocking the King of Thailand, which violated Thailand's laws against insulting the king. Wong visited Thailand and was impressed by the nation's reverence for its monarch, so she reluctantly agreed to block the video — but only for computers located in Thailand.
Since then, selectively banning content by geography — called "geo-blocking" — has become a more common request from governments. "I don't love traveling this road of geo-blocking," Wong said, but "it's ended up being a decision that allows companies like Google to operate in a lot of different places."
For social networks like Facebook, however, geo-blocking is difficult because of the way posts are shared with friends across national boundaries. If Facebook geo-blocks a user's post, it would only appear in the news feeds of friends who live in countries where the geo-blocking prohibition doesn't apply. That can make international conversations frustrating, with bits of the exchange hidden from some participants.
As a result, Facebook has long tried to avoid using geography-specific rules when possible, according to people familiar with the company's thinking. However, it does geo-block in some instances, such as when it complied with a request from France to restrict access within its borders to a photo taken after the Nov. 13, 2015, terrorist attack at the Bataclan concert hall in Paris.
Bickert said Facebook takes into consideration the laws in countries where it
Facebook's goal is to create global rules. "We want to make sure that people are able to communicate in a borderless way," Bickert said.
MySpace had positioned itself as the nightclub of the social networking world, offering profile pages that users could decorate with online glitter, colorful
By comparison, Facebook was the buttoned-down Ivy League social network — all cool grays and blues. Real names and university affiliations were required. Chris Kelly, who joined Facebook in 2005 and was its first general counsel, said he wanted to make sure Facebook didn't end up in law enforcement's crosshairs, like MySpace.
"We were really aggressive about saying we are a no-nudity platform," he said.
The company also began to tackle hate speech. "We drew some difficult lines while I was there — Holocaust denial being the most prominent," Kelly said. After an internal debate, the company decided to allow Holocaust denials but reaffirmed its ban on
"I've always been a pragmatist on this stuff," said Kelly, who left Facebook in 2010. "Even if you take the most extreme First Amendment positions, there are still limits on speech."
By 2008, the company had begun expanding internationally but its censorship rulebook was still just a single page with a list of material to be excised, such as images of nudity and Hitler. "At the bottom of the page it said, 2018Take down anything else that makes you feel uncomfortable,'" said Dave Willner, who joined Facebook's content team that year.
Willner, who reviewed about 15,000 photos a day, soon found the rules were not rigorous enough. He and some colleagues worked to develop a coherent philosophy underpinning the
By the time he left Facebook in 2013, Willner had shepherded a 15,000-word rulebook that remains the basis for many of Facebook's content standards today.
"There is no path that makes people happy," Willner said. "All the rules are mildly upsetting." Because of the volume of decisions — many millions per day — the approach is "more utilitarian than we are used to in our justice system," he said. "It's fundamentally not rights-oriented."
Willner's then-boss, Jud Hoffman, who has since left Facebook, said that the rules were based on Facebook's mission of "making the world more open and connected." Openness implies a bias toward allowing people to write or post what they want, he said.
But Hoffman said the team also relied on the principle of harm articulated by John Stuart Mill, a 19th-century English political philosopher. It states "that the only purpose for which power can be rightfully exercised over any member of a civilized community, against his will, is to prevent harm to others." That led to the development of Facebook's "credible threat" standard, which bans posts that describe specific actions that could threaten
Eventually, however, Hoffman
The rules developed considerable nuance. There is a ban
The documents state that Facebook relies, in part, on the U.S. State Department's list of designated terrorist organizations, which includes groups such as al-Qaida, the
There is also a secret list, referred to but not included in the documents, of groups designated as hate organizations that are banned from Facebook. That list apparently doesn't include many Holocaust denial and white supremacist sites that are up on Facebook to this day, such as a group called "Alt-Reich Nation." A member of that group was recently charged with murdering a black college student in Maryland.
As the rules have multiplied, so have exceptions to them. Facebook's decision not to protect subsets of protected groups arose because some subgroups such as "female drivers" didn't seem especially sensitive. The default position was to allow free speech, according to a person familiar with the decision-making.
After the wave of Syrian immigrants began arriving in Europe, Facebook added a special "quasi-protected" category for migrants, according to the documents. They are only protected against calls for violence and dehumanizing generalizations, but not against calls for exclusion and degrading generalizations that are not dehumanizing. So, according to one document, migrants can be referred to as "filthy" but not called "filth." They cannot be likened to
Facebook also added an exception to its ban against advocating for anyone to be sent to a concentration camp. "Nazis should be sent to a concentration camp," is allowed, the documents
The Facebook policy appears to have had repercussions in many of the at least two dozen disputed territories around the world. When Russia occupied Crimea in March 2014, many Ukrainians experienced a surge in Facebook banning posts and suspending profiles. Facebook's director of policy for the region, Thomas Myrup Kristensen, acknowledged at the time that it "found a small number of accounts where we had incorrectly removed content. In each case, this was due to language that appeared to be hate speech but was being used in an ironic way. In these cases, we have restored the content."
Katerina Zolotareva, 34, a Kiev-based Ukrainian working in communications, has been blocked so often that she runs four accounts under her name. Although she supported the "Euromaidan" protests in February 2014 that antagonized Russia, spurring its military intervention in Crimea, she doesn't believe that Facebook took sides in the conflict. "There is war in almost every field of Ukrainian life," she says, "and when
In Western Sahara, a disputed territory occupied by Morocco, a group of journalists called Equipe Media say their account was disabled by Facebook, their primary way to reach the outside world. They had to open a new account, which remains active.
"We feel we have never posted anything against any law," said Mohammed
In Israel, which captured territory from its neighbors in a 1967 war and has occupied it since, Palestinian groups are blocked so often that they have their own hashtag, #FbCensorsPalestine, for it. Last year, for instance, Facebook blocked the accounts of several editors for two leading Palestinian media outlets from the West Bank — Quds News Network and
Last year India cracked down on protesters in Kashmir, shooting pellet guns at them and shutting off cellphone service. Local insurgents are seeking autonomy for Kashmir, which is also caught in a territorial tussle between India and Pakistan. Posts of Kashmir activists were being deleted, and members of a group called the Kashmir Solidarity Network found that all of their Facebook accounts had been blocked on the same day.
Ather Zia, a member of the network and a professor of anthropology at the University of Northern Colorado, said that Facebook restored her account without explanation after two weeks. "We do not trust Facebook
Facebook users who don't mince words in criticizing racism and police killings of racial minorities say that their posts are often taken down. Two years ago, Stacey Patton, a journalism professor at historically black Morgan State University in Baltimore, posed a provocative question on her Facebook page. She asked why "it's not a crime when White freelance vigilantes and agents of 2018the state' are serial killers of unarmed Black people, but when Black people kill each other then we are 2018animals' or 2018criminals.'"
Although it doesn't appear to violate Facebook's policies against hate speech, her post was immediately removed, and her account was disabled for three days. Facebook didn't tell her why. "My posts get deleted about once a month," said Patton, who often writes about racial issues. She said she also is frequently put
"It's such emotional violence," Patton said. "Particularly as a black person,
Didi Delgado, whose post stating that "white people are racist" was deleted, has been banned from Facebook so often that she has set up an account on another service called Patreon, where she posts the content that Facebook suppressed. In May, she deplored the increasingly common Facebook censorship of black activists in an article for Medium titled "Mark Zuckerberg Hates Black People."
Facebook also locked out Leslie Mac, a Michigan resident who runs a service called SafetyPinBox where subscribers contribute financially to "the fight for black liberation," according to her site. Her offense was writing a post stating "White folks. When racism happens in public — YOUR SILENCE IS VIOLENCE."
The post does not appear to violate Facebook's policies. Facebook apologized and restored her account after TechCrunch wrote an article about Mac's punishment. Since then, Mac has written many other outspoken posts. But, "I have not had a single peep from Facebook," she said, while "not a single one of my black female friends who write about race or social justice
"My takeaway from the whole thing is: If you get publicity, they clean it right up," Mac said. Even so, like most of her friends, she maintains a separate Facebook account in case her main account gets blocked again.
Negative publicity has spurred other Facebook turnabouts as well. Consider the example of the iconic news photograph of a young naked girl running from a napalm bomb during the Vietnam War. Kate Klonick, a Ph.D. candidate at Yale Law School who has spent two years studying censorship operations at tech companies, said the photo had likely been deleted by Facebook thousands of times for violating its ban on nudity.
But last year, Facebook reversed itself after Norway's leading newspaper published a front-page open letter to Zuckerberg accusing him of "abusing his power" by deleting the photo from the newspaper's Facebook account.
n December 2015, a month after terrorist attacks in Paris killed 130 people, the European Union began pressuring tech companies to work harder to prevent the spread of violent extremism online.
After a year of negotiations, Facebook, Microsoft,
Now the German government is considering legislation that would allow social networks such as Facebook to be fined up to 50 million euros if they don't remove hate speech and fake news quickly enough. Facebook recently posted an article assuring German lawmakers that it is deleting about 15,000 hate speech posts a month. Worldwide, over the last two months, Facebook deleted about 66,000 hate speech posts per week, vice president Richard Allan said in a statement Tuesday on the company's site.
Among posts that Facebook didn't delete were Donald Trump's comments on Muslims. Days after the Paris attacks, Trump, then running for president, posted on Facebook "calling for a total and complete shutdown of Muslims entering the United States until our country's representatives can figure out what is going on."
Candidate Trump's posting — which has come back to haunt him in court decisions voiding his proposed travel ban — appeared to violate Facebook's rules against "calls for exclusion" of a protected religious group. Zuckerberg decided to allow it because it was part of the political discourse, according to people familiar with the situation.
However, one person close to Facebook's decision-making said Trump may also have benefited from the exception for sub-groups. A Muslim ban could be interpreted as being directed against a sub-group, Muslim immigrants, and thus might not qualify as hate speech against a protected category.
Hannes Grassegger is a reporter for Das Magazin and
ProPublica is a Pulitzer Prize-winning investigative newsroom.