During the 2016 presidential elections Bill Clinton believed his wife’s team had failed to learn one of the lessons of Brexit: working people felt alienated and there was an anti-establishment mood in the air. But his suggestions were politely acknowledged and then discreetly shelved by the number-crunchers in charge. “He’d report back from the field on what he was hearing at campaign events,” Jonathan Allen and Amie Parnes wrote in Shattered: Inside Hillary Clinton’s Doomed Campaign. “[The] response was always a variation on the same analysis: the data run counter to your anecdotes. Bill liked data, but he believed it was insufficient … He felt it was important to talk to voters and get a real sense for what they were feeling.” In the battle between raw numbers and raw emotion, the numbers won and Hillary lost.

Recent revelations that Donald Trump’s campaign paid Cambridge Analytica to target potential voters with bespoke messages would lead some to argue that Hillary’s problem wasn’t that her team privileged data over lived experience. The problem was that Team Trump had better data.

In an undercover recording for Channel 4 Cambridge Analytica’s head of data, Alexander Tayler, effectively claimed Trump owed his victory to his company: “When you think about the fact that Donald Trump lost the popular vote by 3m votes but won the electoral college vote,” he bragged. “That’s down to the data and the research.”

The still unfolding story of how personal information was leveraged for financial profit and political gain lays bare the vulnerability of democracy to new technology. The situation was bad enough. With social media lies can spread far faster than scrutiny can ever travel because people are now able to curate both their own news and the communities in which they distribute it. This is, partly, why 12 years after the start of the Iraq war, half of Republicans still believed Iraq had weapons of mass destruction (it didn’t) and two years ago 59% of Trump supporters still thought Barack Obama was not born in America (he was). Ignorance and bigotry need no help: in the fetid pit of prejudice and presumption they can breed and disperse freely. So to have private contractors harvesting personal information and using it to target the people they think are most susceptible to certain political messages, all for electoral gain and corporate profit, is disturbing. Facebook, which stands accused of failing to monitor how data collected from the platform was being used, should be held to account; Cambridge Analytica, which was allegedly linked to the leave side in the Brexit referendum , should also be investigated.

The questions raised in the past week about the harvesting of data, and the targeting of voters based on personal information that they may have parted with unknowingly, are serious. And yet to accept the claim that outside intervention was primarily responsible for Trump’s victory or the Brexit vote feels like an even more serious transgression.

There is a desire among some liberals, many even, to dismiss the events of recent years as something other than what they were – a massive political defeat. Some of this dismissiveness comes from a deeply illiberal place, that effectively insists people are too stupid to understand what they want or what is best for them. There is also a wish to avoid responsibility for failing to connect politically in the decades since the financial crisis with a message that resonated, and to seek instead a legal remedy or technical reprieve by impeaching the president or annulling the referendum.

There is no contradiction between accepting that the Russians, Facebook, ignorance, Cambridge Analytica or stupidity played a role and understanding that a far larger drama, which has been under way for far longer, makes sense of the role they were able to play. If Trump is removed by impeachment, so be it. But that will be an inadequate salve for the long-running sores that made him possible.

Cambridge Analytica didn’t invent the racism, misogyny and xenophobia that have corroded American political culture and made Trump possible. These social pathologies were eating away at our democracies long before social media was invented. Their roots are deeper, their reach farther and their influence greater than any algorithm could conjure.

The Russians did not tell Hillary Clinton to send surrogates to Arizona shortly before the election, when they would have been far more helpful in Michigan or Wisconsin. They didn’t tell her to give a speech at Goldman Sachs for $225,000 in the wake of a huge bank bailout or call half Trump’s supporters a “basket of deplorables”. Facebook hadn’t been invented when she was for the Iraq war before she was against it, or when she was against gay marriage before she was for it.

Social media platforms, nefarious bots and dodgy data companies can amplify views that are already out there and distort perspectives that already exist – but they cannot invent them. In short they can manipulate the supply of ideas, but they cannot create the demand for those ideas from thin air.

Trump’s victory and Brexit came in a time of growing economic inequality, wage stagnation, mass migration, rising xenophobia and racial tension. Bill Clinton was right – there was a mood out there. That mood emerged from material conditions that mainstream parties had made possible over the past few decades, through, among other things, illegal war, deregulation, trade liberalisation and vile rhetoric on immigration. And on both sides of the Atlantic liberals failed to either take responsibility for this or adequately respond to its consequences. Now they are paying the price.

None of this excuses the actions of those who have sought to mislead us nor denies the power of the memes, messages and misinformation that they have produced. With some help, what begins life as received wisdom can graduate to common knowledge and mature into a full-blown script that people recite without knowing where it came from or what, precisely, it means. “We just put information [like ‘crooked Hillary’] into the bloodstream of the internet and watch it grow,” claimed Cambridge Analytica’s CEO, Alexander Nix, who has since been suspended. “Give it a little push every now and again over time to watch it take shape … So this stuff infiltrates the online community with no branding, so it’s unattributable, untrackable.”

We know these things made a difference. Given the narrow margins of defeat in both the Brexit referendum and the 2016 presidential election, it is an open question whether they made the difference that tipped us over the edge. That matters.

But either way, the question remains: what were we doing so close to the edge? And how do we get out of the pit we are now in? We do not yet have the answer to the second question. But we can be fairly sure we won’t find it in an algorithm.

• Gary Younge is editor-at-large for the Guardian