Politics and Profit on Social Media

In the Habit of Hate

How Social Media Profits from Radicalization & Mass Harassment.

Harvard Law Student

April 24, 2025

You wake up to your screeching alarm, curtains drawn and room dark. Rolling around blindly in bed, you begin fumbling for your phone on sound alone. The light hits your face and you squint a little, and as your eyes adjust to the first bit of light you’ve seen that day, you begin to look through your notifications from Facebook, Twitter, and Truth Social.

Without really thinking about it, you tap on Twitter, checking on the activity of your favorite account, Libs of TikTok. You scroll through recent posts and see that while the recent efforts against Planet Fitness seem to have had some effect on the company’s bottom line, Raichik reports biological men are still allowed in locker rooms:

“I just received this image from Planet Fitness in Alaska. The man who was caught shaving in the womens’ bathroom is still entering womens’ private spaces . . . . Planet Fitness is not safe for women!!”

You can see his face in the photo. It’s time for action.

Harassment and death threats sent to an election volunteer’s home.

Violent messages and phone calls against Boston Children’s Hospital employees.

Threats of planted bombs at more than 2 dozen public schools and libraries, evacuating students and shutting down classes.

More than 54 bomb threats to Planet Fitness.

Meanwhile, purveyors of accounts like Libs of TikTok earn Twitter more than 6.4 million dollars yearly.

This is merely a taste of the trail of violence—and profit—that follows in the wake of just one social media account engaged in hate politics. This sort of violence, called networked mass harassment, has surged in recent years.

While it might be tempting to treat these accounts and their followers as outliers, there is a broader, structural story to be told here: in a world where the average adult will now spend 17 years of our lives online, how easy might it be for us to find ourselves subtly pushed into spaces like these by social media? In a country with a crisis of loneliness, how easy might it be to self-medicate with online hate politics?

And most importantly, why hasn’t anyone done anything to stop this?

A far-right group organizes around the State House in Boston, carrying flags and signs. A few notable signs read “Close the Healey Hotels,” “Put Alligators in the Rio Grande River,” “Veterans Before Illegals,” “Mayor Wu is a Communist Dictator,” and “All Lives Matter.”

A far-right rally protests outside of the state house in Boston. By Samantha Slack.

The answers to these questions are troubling. Evidence overwhelming demonstrates that social media preferences controversial and often negative content for profit, a ripe opportunity for radicalization.

Simultaneously, social media platforms and political figures are disincentivized from acting on these harms, whether by fear, profit or political gain. Instead, platforms advance contradictory narratives to discourage legal liability for harms, while maintaining their competency to self-regulate.

All the while, the gap in legal recourse for victims of social media harms has grown. The courts have been gagged on the sidelines, in large part due to Section 230 of the Communications Decency Act, which provides complete liability protections to platforms for the content that they host, as well as their moderation of it. Nonetheless, the courts are beginning to stir, and scholars and activists are continuing to look for new and creative ways to bring the law back into social media.

Engagement At All Costs

Social media platforms profit, despite ostensibly offering free services, from two main avenues: advertisements and user data. Both measures rely upon engagement with the platform; more engagement provides more opportunity to advertise to the user and more opportunity to gather information on the user’s preferences for sale to advertisers. As such, it is distinctly in a platform’s best interest to keep users returning to the platform.

So, the question then becomes, what keeps users engaged? One answer has to do with which users are vulnerable to social media’s positive stimulus.

Alexa Cilia, a post-baccalaureate researcher with UNC Chapel Hill’s Winston Family Initiative in Technology and Adolescent Brain Development (WiFi), revealed that “hyper responsivity to positive social feedback in four different brain regions commonly associated with social information processing, may represent a risk factor for addiction-like social media use in later adolescence.” In other words, “if someone, let’s just say, is more sensitive to social positive feedback [or rewards,] that might be associated with [habitual] social media use.”

However, there is a more disturbing answer from the inside: Facebook’s own research teams reported in 2018 that their “algorithms exploit the human brain’s attraction to divisiveness.” Social media platforms breed engagement with content that evokes and networks negative emotions in the user.

Part of this comes from dark aspects of human psychology. An NYU study measured the reach of half a million tweets and found that moral and emotional words compounded virality by roughly 20 percent.

Algorithms exploit the human brain’s attraction to divisiveness.

Further research in 2021 found that the largest predictor of virality was actually the out-group effect: “each individual term referring to the political out-group increased the odds of a social media post being shared by 67%.” It was through othering language—especially political othering language—that posts were most likely to evoke and anger, and therefore go viral.

The propping up of this content is not just an unfortunate side effect of those using the platforms: it is by design. As explained by Cathy O’Neil in her books Weapons of Math Destruction and The Shame Machine, social media preferences content which evokes a shock response in users, like mass misinformation and conspiracy theory. O’Neil characterizes social media platforms as “networked shame engines” which seek to profit from hateful engagement, propping up content—and often people—for users to “bombard with digital tomatoes.”

Radicalization by Algorithm

What is unfortunate then, is the disturbing alignment between social media’s business model and the path to radicalization. Social media’s dark psychology underbelly—the prioritization of othering and controversial content, the creation of echo chambers, the targeted content—all provide tools for the process of radicalization.

Radicalization tends to follow a straightforward, well-documented path that online researchers have termed “The Alt-Right Playbook.” It begins with identifying the vulnerable audience member—especially a lonely one—and offering them community.

Another researcher from UNC Chapel Hill, Dr. Nathan Jorgensen, offered insights into how this happens on the ground. Jorgensen, who has studied white racial identity development, notes that in a culture of such staunch divide around race, white men often feel that they are told “you’re the bad guy,” which can leave them with difficult feelings to process. Jorgenson notes that, “when the only counter-narrative is extreme, that’s when people get radicalized. Conservative media is able to validate that feeling . . . and then give them a narrative.”

An older white man at a political rally frowns into his tri-pod, live-streaming his thoughts to social media.

An attendee to a right-wing political rally live-streams to his followers. By Samantha Slack.

This initial draw is quickly followed by establishing shared spaces where “the ideology is the price of community.” Such shared spaces include message boards, restricted online spaces, and the ubiquitous Facebook group. Especially in social media platforms like Facebook, X, and Reddit that encourage continuous engagement, the message board becomes a habitual part of one’s everyday life, and soon the target is immersed.

“There is an echo-chamber effect, especially with social media, the more we search for something, the more we receive or are sent this information similar to it . . . . This normalizes it, and desensitizes it. Social media hate discourse increases social polarization, gives a feeling of legitimacy and normalcy to hateful rhetoric, and people can do it anonymously.”

The final step lies in isolation of the subject. Some of this happens naturally in social media’s echo chambers. Dr. Ghayda Hassan, a leading researcher on radicalization from the Universite de Quebec au Montreal, notes, “There is an echo-chamber effect, especially with social media, the more we search for something, the more we receive or are sent this information similar to it . . . . This normalizes it, and desensitizes it. Social media hate discourse increases social polarization, gives a feeling of legitimacy and normalcy to hateful rhetoric, and people can do it anonymously.”

The Other Side of the Screen: Networked Mass Harassment’s Real-World Effects

To understand a little bit more of what it can be like on the other side of the screen, I spoke with Alejandra Caraballo. Ms. Caraballo is a civil rights attorney working at Harvard Law School’s Berkman Klein Center, a trans advocate, and an avid social media user. In recent years, Ms. Caraballo’s work has focused on networked mass harassment against the LGBTQ community and private individuals, especially by Libs of TikTok.

Libs of TikTok is ironically neither a liberal nor a TikTok account. Active on X and Facebook, the account is run by Chaya Raichik, a conservative activist who reposts videos of liberal TikTok accounts with commentary. “Starting in late 2021 into 2022, [Raichik] realized that she was getting massive engagement on anti-LGBTQ content. [She] was instrumental in pushing the groomer libel and her account exploded, went from less than a few hundred thousand followers to now, in the last 2 years, 3 million followers.” Raichik began to see mainstream success: the account “was promoted heavily by people in the conservative movement, by Fox News, and by others” like Tucker Carlson.

Inextricable from Raichik’s success has been the violence that follows in her wake. Caraballo notes that she employs a “very targeted harassment of individuals, and incited threats and harassment follow [Raichik’s posts]. [There are], just at least, 30 instances of bomb threats tied to her tweets.”

Accounts of this kind have a distinct impact in the online space, especially for those targeted. Caraballo notes, “Her tweets actually chill other people[’s speech] because she has a massive platform . . . and the people who are subjected to her tweets get so inundated with harassment and threats that they lock down, they delete their accounts.”

We face a clandestine and “quiet erosion of our democratic institutions.

These enemies are not limited to the online space either. “These kinds of accounts are not just targeting LGBTQ people, they’re targeting election officials, they’re targeting public health officials, they’re targeting school board members… The victims are just the average typical person who’s just volunteering to be an election worker, or just a middle-class parent who like, is also on the school board, has a full-time day job, and is getting inundated with death threats.”

What is the rational response for the victim then? “They’re going to not run for re-election. They’re going to resign and just say, like, I don’t want to deal with this . . . . [This happens] in a way that intimidates people from participating in democratic institutions.” Caraballo warns we face a clandestine and “quiet erosion of our democratic institutions. [Networked mass harassment is] hollowing them out.”

The Moderation Microcosm

While Libs of TikTok has faced suspension on Facebook previously, X has consistently under-moderated the account. This comes despite multiple flagrant references by Raichik to bomb threats, including changing her profile picture and background to reference the matter

Raichik, smiling, poses with a newspaper article detailing the bomb threats, hate mail, and violence that follow in her wake.

In fact, Libs of TikTok, Elon Musk (prior to his acquisition of X), and President Donald Trump engaged in a campaign directly against Twitter’s former Head of Trust and Safety, Yoel Roth, in 2019. After fact-checking one of Trump’s tweets, Roth recounted that “Legions of Twitter users, most of whom days prior had no idea who I was or what my job entailed, began a campaign of online harassment that lasted months, calling for me to be fired, jailed or killed.” Roth believes it was part of a strategy to discourage platforms from moderating Mr. Trump—decisions that culminated in violence at the Capitol on January 6th.

This is just one example of concerning pattern in content moderation: platforms are extremely slow to moderate those with powerful influence. For example, Twitter has failed to act on 99% of Twitter Blue, or paid subscriber, accounts tweeting out hateful content. The Center for Countering Digital Hate maintains the platform is “allowing them to break its rules with impunity and is even algorithmically boosting their toxic tweets.”

This is true internationally as well. Facebook’s lack of content moderation has been implicated in the Rohingya massacre in Myanmar; the platform declined to censor anti-Rohingya content, allegedly because it did not have Burmese-speaking content moderators on staff. In a reflection of societal structures of power, Caraballo notes that “people who are vulnerable, and people who don’t have political power in their colleagues, are the ones who then get hit disproportionately in moderation decisions.” Content moderation systems are a microcosm of power.

Powerless or Powerful? Platform’s Competing Narratives of Decentralization and Self-Regulation

Social media websites rely upon strong PR and lobbying teams to maintain the status quo and evade a regulatory crackdown. However, a close examination of the varying stories coming from the industry uncovers a number of inherent contradictions: these websites simultaneously portray themselves as powerless to control the decentralized content production on their sites, while advocating for self-regulation to resolve nascent harms.

The first weapon in the industry’s narrative arsenal is attributing social media harm—especially extremism—to users hijacking the platform for their own interests. In cases of social media extremism, there are clear actors that sites can point to abusing the platform—users hidden among millions. For example, in oral arguments in Google v. Gonzalez, counsel for Google was very clear in asserting the difference between the platform providing the content and third-party users. In the face of massive volumes of material produced each day, social media websites, if liable for all harmful content, could be litigated out of existence.

A figure stands in the shadow wearing a hoodie. His face is impossible to make out.

Platforms place blame on website users—oft-impossible to identify—for the proliferation of harmful content. By Kaique Rocha, free license.

Another specter haunts content regulation: overbroad censorship. Legal theorists have engaged in extensive debates on the First Amendment concerns that underly content moderation decisions. But even taking for granted that no First Amendment rights exist for users, platforms engaging in content moderation cite interests in protecting free expression. “People having the power to express themselves at scale is a new kind of force in the world. It is a Fifth Estate alongside the other power structures of society,” said Zuckerburg in 2019.

Who might hold the required expertise then? According to many social media sites, themselves.

These concerns create a Catch-22 that thoroughly discourages public law regulation. Public regulation simultaneously appears to threaten over-enforcement and under-enforcement: “Regulation incentivizing companies to focus on removing the subset of proscribed content that is illegal . . . . are likely to produce over-enforcement on legitimate internet users that highlight political or cultural issues . . . . [and yet] if exposed to liability for identifying such threats and referring them to law enforcement, platforms may be disincentivized from doing so.”

Platforms and legal theorists also rely on technological exceptionalism and the narrative of a lack of expertise to attack legislative and regulatory efforts.

Who might hold the required expertise then? According to many social media sites, themselves. In a complete about-face from the decried powerlessness these sites use to discourage liability regimes, they platforms uphold self-regulation as the preeminent solution.

Facebook, for example, has created an Oversight Board (oft-nicknamed Facebook’s “Supreme Court) to make the most difficult content moderation decisions on the platform. The platform sought out “sage advisors,” which include humanitarian activists and even a Nobel Laureate, to fill the seats of its justices, who are rotated out for cases to ensure variety on the bench. The cases proceed with pomp and circumstance, with justices requesting amicus brief-style comments from interested parties and publishing their decisions.

Legal theorists like Bowers and Zittrain have celebrated the procedural justice of the Oversight Board system as a move away from the quagmire of rights narratives and into a “professionalization” of content moderation. With firm procedure can come legitimacy.

Positively, Oversight Board decisions are binding upon the platform. However, the Board is limited in its ability to take on more than a handful of cases per year (around 20), just as the Supreme Court might be. The cases are also highly fact-specific. As such, the Board cannot influence the system of content policy enforcement proactively, except though non-binding recommendations.

Brenda Dvoskin, a doctoral candidate at Harvard Law School, has written extensively on the Oversight Board’s design and procedural founding in Human Rights law. However, she notes in her piece Expert Governance of Online Speech, however or undesirable this result may be, that “Ultimately, this governance model shifts power to experts under the illusion of empowering the people.”

Political Capture of Social Media?

Further contributing to the problem, the legal theorists and researchers I interviewed expressed serious concerns about elected politicians’ surreptitious interests in narratives on social media.

Ms. Caraballo reports that it is precisely the political allies of Ms. Raichik that make it so difficult to moderate Libs of TikTok: “[moderators] are scared of her and they’re also scared of their own vocations . . . because she has so many powerful political allies. Acting on this account and moderating this account is a political problem. It’s not just a safety problem.”

“Right now, they’re more scared of being called in front of Congress and questioned about moderation decisions than they are about being asked about [bomb threats.]”

The interests that politicians might have in using social media to shape the mind of the polity are ever-present in discussions of content moderation. While Gilad Mills, a doctoral student at Harvard Law and associate of the Berkman Klein Center, recognizes the importance of public law intervention, he has serious concerns:

“For now, I think allowing state intervention in content moderation is going to be very problematic . . . . [In part because of the political process, but also,] it could be used to kind of start pushing platforms towards specific results. Results a specific legislature or specific, government would prefer. So, everything should be very careful.”

Mills is not alone. David Greene, a senior staff attorney at the Electronic Frontier Foundation, maintains that “content moderation as a process is really problematic. [But,] I don’t think the right solution to that is to give the government the ability to impose its own editorial viewpoints on private actors. I think that’s a dangerous power to hand the government.”

A photo of the Supreme Court building in Washington, D.C.

The Supreme Court is poised to hear arguments on state regulation of social media in 2024. By Phuong Touch, free license.

Nonetheless, in 2021 both Florida and Texas’ state legislatures passed bills intended to “stop social media giants from throttling conservative views.” While the laws have different timbres—one banning viewpoint discrimination by platforms, and the other restricting suspension of political figureheads’ accounts—implementation of both has been stopped by preliminary injunction. The decision on their legality is now in the hands of the Supreme Court.

A Path Paved on Broken Promises

Historically, courts have been highly reluctant to wade into the murky waters of content moderation. This fear has produced a gaping hole in private law means to pursue compensation from social media companies for harms to users. At the heart of this reluctance is Section 230 of the Communications Decency Act:

“No provider or user of an interactive computer service shall be treated as the publisher or speaker of any information provided by another information content provider.”

Often considered the 26 words that created the Internet, Section 230 was passed in the 1990s to protect the burgeoning tech industry from liability for third-party content generated by its users. In other words, it prevents lawsuits against internet platforms for content moderation, or the lack thereof.

But, despite being a shield against liability, social media platforms have weaponized Section 230 in court. A highly powerful tool at the pleading stage, Section 230 allows defendant platforms to quickly brush away claims based on user posts.

“No provider or user of an interactive computer service shall be treated as the publisher or speaker of any information provided by another information content provider.”

Because of the power of Section 230, Caraballo believes that direct defamation suits against harassing accounts is one of the only ways forward for victim plaintiffs. However, the process is costly and uncertain, and provides no structural recourse for patterns of radicalization or mass harassment.

Researchers like Gilad Mills have been puzzling through ways to bridge this very gap. Throughout his doctorate, he has been exploring unique approaches that avoid harming free speech rights while offering the worst victims of social media’s broken promises a means for recourse. In his forthcoming article, A Contractual Approach to Social Media Governance, Mills dives into a unique path forward.

A scraggly mountain path disappears into mist

Forging a new interpretation of Section 230 may be an uphill battle. By Jim Richter, free license.

Mills and other legal theorists believe that at the heart of the business exchange between users and platforms is a simple trade. Users offer engagement and data. Platforms offer—no, not just use of their platform—but “platforms’ best-efforts commitments” to content moderation. Therefore, Mills proposes a contractual approach to making platforms hold up their end of the bargain.

“We have been trapped in dichotomies for too long . . . . Public and private [law] have always, from time immemorial, been working in harmony . . . . It’s both, it’s always been both, and it shouldn’t have stopped, even after Section 230.”

Mills believes that Section 230 was passed with “a very clear purpose to prevent tort liability . . . [but that] contract liability is very different.” And platforms have made big promises.

Mills hopes that judges “trying to find solutions after having been out of the game for 30 years” look beyond the four corners of the Terms of Service to understand the bargain that platforms have made with users:

“community guidelines, data policies, and other documents provided by the platform elaborate on the contents of contractual norms and obligations, while the Terms of Service agreement simply defines those contents as obligatory.”

By adopting an expansive definition of the contract between platform and user, Mills hopes that consumer contract law can adapt to force platforms to enforce their community guidelines, keeping their promises to users.

“We have been trapped in dichotomies for too long . . . . Public and private [law] have always, from time immemorial, been working in harmony . . . .It’s both, it’s always been both, and it shouldn’t have stopped, even after Section 230.”

Political radicalization on social media has begun to present profound problems for our everyday safety, for our mental health, and for the legitimacy of our political institutions. Truly independent self-regulation of social media remains uncommon, and political solutions are mired in controversy. In the face of such acute difficulties, the courts can no longer remain silent.

Harvard Law Student

Editor’s Picks

Social Media Harms