My eye was caught by a headline in Wired magazine: “When algorithms think you want to die”. Below it was an article by two academic researchers, Ysabel Gerrard and Tarleton Gillespie, about the “recommendation engines” that are a central feature of social media and e-commerce sites.

Everyone who uses the web is familiar with these engines. A recommendation algorithm is what prompts Amazon to tell me that since I’ve bought Custodians of the Internet, Gillespie’s excellent book on the moderation of online content, I might also be interested in Safiya Umoja Noble’s Algorithms of Oppression: How Search Engines Reinforce Racism and a host of other books about algorithmic power and bias. In that particular case, the algorithm’s guess is accurate and helpful: it informs me about stuff that I should have known about but hadn’t.

Recommendation engines are central to the “personalisation” of online content and were once seen as largely benign. Sure, they tended to lock people into their own filter bubbles and might therefore contribute to political polarisation or fragmentation, but on the other hand, they were giving people what they wanted, so where’s the harm in that?

Quite a lot, as it turned out, but it took us a while to realise that. Gerrard and Gillespie open their article by revisiting the tragic case of Molly Russell, the young teenager who sought out images of self-harm before she took her own life in 2017. It was later discovered, the authors report, “that these images were also being delivered to her, recommended by her favourite social media platforms. Her Instagram feed was full of them. Even in the months after her death, Pinterest continued to send her automated emails, its algorithms automatically recommending graphic images of self-harm, including a slashed thigh and cartoon of a young girl hanging.”

In the wake of scandals and tragedies such as Molly Russell’s death, social media companies are regularly excoriated for “hosting” such material on their sites. (They’re actually publishing them, but a legal loophole allows them to pretend that they’re not and thereby escape legal liability.) Faced with rising public concern, the companies have ramped up their “moderation” (ie censorship) operations, maintaining large teams of contractors to monitor and, if necessary, remove offending content from their sites. Given the scale of online content, this task makes that of Sisyphus look like a doddle, but at the moment the companies are in denial about that. In the meantime, many of the people contracted to carry out the impossible task continue to be traumatised by it.

The sad truth about the internet is that it holds up a mirror to human nature and some of what we see reflected in it is pretty repulsive. (One of the videos reportedly used in training sessions for Facebook moderators in Arizona, for example, shows a man being murdered. “Someone is stabbing him, dozens of times, while he screams and begs for his life.”) The social media companies, which like to portray their mission as enabling people to express themselves (“Broadcast yourself” was an early motto of YouTube, for example), protest that they can hardly be blamed if people are horrible, express themselves in unacceptable ways and generally foul the online nest.

Their reliance on recommendation engines rather undermines this comforting narrative. As Gerrard and Gillespie point out: “Social media platforms not only host this troubling content, they end up recommending it to the people most vulnerable to it. And recommendation is a different animal than mere availability.”

Spot on. And there’s a growing body of evidence to support this. In areas such as self-harm, misinformation, terrorist recruitment or conspiracy theories, social media platforms don’t just make this content easy to find – their recommendation engines help to amplify it.

YouTube’s recommender, for example, sometimes seems to boost conspiracy theories and extremist content, to the point where (as the “alt-right” discovered years ago) it might be the most powerful “radicaliser” on the planet.

If recommendation engines are indeed significant generators of online harms, what might be done about them? One approach involves trying to make them less crass and more sensitive to context. Just because somebody suffering from depression searches for information about the condition doesn’t mean that they’re interested in suicide, for example. Improving the inferential processes of software is not easy, but it’s feasible, given the will and resources to do it.

“Given the will...”. The bigger problem is that recommendation engines exist for a reason – to keep users on a site and increase their “engagement” with its content. The surest way to keep people engaged is to lure them down wormholes filled with intriguing things.

Which is precisely what these engines are good at – and why social media companies won’t give them up. Nobody who ever had a golden goose voluntarily wrung its neck.

What I’m reading

Digital footprint
Wired magazine’s story about how Paul Manafort couldn’t cover his illegal tracks: tech-savvy he ain’t.

Word on a wing
How a text message saying “I love you” gets to its destination. Motherboard’s truly beautiful explanation of the everyday miracle that is SMS.

Gridlock ahead
Think that self-driving cars will reduce traffic? Think again, says this research paper from the University of California, Santa Cruz.

• Comments on this piece are premoderated to ensure discussion remains on topics raised by the writer. Please be aware there may be a short delay in comments appearing on the site.