Why has David Moyes had such a horror show since taking over as Manchester United manager last summer? From our armchairs, the diagnosis has been relatively straightforward: taking over from a legend is inevitably a fool's errand; anyone replacing Sir Alex Ferguson was doomed before a ball was kicked. Moyes inherited a patchy squad with too few players at the peak of their powers. Or, if you want to be snarky, you might query Moyes's credentials: he never won a major trophy as a manager at Preston North End and Everton and has now brought a smaller-club mentality to United, arguably the most famous football organisation on the planet.

Moyes clearly has a different perspective on the crisis. While he is restricted to bringing in new players by two transfer "windows" – one over summer, the other during January – he can make changes to personnel behind the scenes whenever he likes. At the end of last year he overhauled United's back-room staff. The arrivals included Robbie Cooke, Everton's chief scout; Chelsea's European scout Mick Doherty, who also worked with Moyes at Everton; and John Murtough, formerly responsible for Everton's vaunted academy and latterly the Premier League's head of elite performance. His final "transfer" was James Smith, head of technical scouting at Everton.

None of these appointments made headlines, but Moyes believes they could be crucial in unearthing the future stars of Manchester United – within the club and outside – and turning round his fortunes at Old Trafford.

There has been a revolution in football – though it is one that even the most committed fans will only be dimly aware of. Clubs are becoming smarter, more efficient. We've probably all seen the graphics and statistics that pop up in newspapers and on shows such as Match of the Day: it began with counting corners and shots on goal, but recently the analysis has become more whizz-bang; not least speed profiling and heat maps, which plot a player's movement around the pitch. But this is just a fraction of the data that can be collected during a match. Opta, a sports statistics company, records around 1,500 "events" from every fixture.

All 20 clubs in the Premier League – and many in the lower divisions – now employ data analysts to make sense of this information. Manchester City has 11 of them. In 2012, Liverpool caused a stir by creating a new position, director of research, for Ian Graham, who has a PhD in theoretical physics. The analysts are involved in pre-match preparation and post-game debriefs; they help to identify transfer targets and devise strategies for nurturing young players through the ranks. These developments have inspired confusion and even suspicion from many supporters, summed up by a recent headline in the New Statesman: "How the spreadsheet-wielding geeks are taking over football."

We can't be blamed for being perplexed. Take the match last month between Arsenal and Bayern Munich, which Bayern won 2-0. The following morning, the Guardian plucked out two statistics: Toni Kroos, the German midfielder, completed more passes than the entire Arsenal midfield; meanwhile, Arsenal's Mesut Özil covered 11.69km, the third-highest distance on the pitch. What the stats didn't say, but was blindingly obvious to anyone watching, was that Kroos was sensational and Özil had a stinker.

These are simplistic examples, but they encapsulate a debate taking place at the highest levels of many football clubs. In one corner are the "quants" or quantitative analysts: they are admirers of the statistician and election-oracle Nate Silver; the Nobel prize-winning psychologist Daniel Kahneman; and especially Billy Beane, the star of Moneyball, Michael Lewis's 2003 book about the data revolution in baseball. They believe that a football match can be translated into numbers and – much as a hedge-fund trader does with the stock market – those figures can be crunched and scanned for patterns. They don't think intuition should be removed from the game but they have found that statistics are dispassionate in a way that humans never are.

As Beane, general manager of the Oakland A's, has said: "The idea that I [should] trust my eyes more than the stats, I don't buy that because I've seen magicians pull rabbits out of hats and I know that the rabbit's not in there."

In the other corner are the traditionalists, which is to say the owners and managers of the overwhelming majority of professional football clubs. They are aware of Moneyball – at least the film starring Brad Pitt – but don't believe the lessons of a stop-start sport such as baseball can be applied to the fluid dynamics of a football match. Most managers once played the game themselves at a high level and it is this fact, they contend, that gives them a special insight into what happens on the pitch and which players they recruit. This approach is summed up by an anecdote about Harry Redknapp, reported in Wired magazine. When he was manager of Southampton, he turned to his analyst after a loss and said: "I'll tell you what, next week, why don't we get your computer to play against their computer and see who wins?"

It turns out that Redknapp was not too wide of the mark: how long will it be before we look at football not just as a contest between 22 players or a clash between two managers, but as a battle between the respective brains trusts assembled on the two benches?

A decent place to start the investigation is Everton FC. As Simon Kuper, the Financial Times columnist and co-author of Soccernomics has detailed, no club in the Premier League has so consistently overachieved during the past decade. Under Moyes, they finished eighth or higher every season from 2007 to 2013. They've managed this despite being more frugal with wages than all of their rivals and not splashing cash on big-name transfers. Instead they achieved success by developing brilliant home-grown talent – Wayne Rooney, Jack Rodwell and Ross Barkley among them – and melding these players with unheralded stalwarts such as Leighton Baines and Leon Osman, who just happen to be statistical outliers.

Baines, in fact, is something of an emblem for the data revolutionaries. For years, he was a solid, dependable left-back with an anachronistic mop-top, a perennial understudy to the flashier Ashley Cole in the England team. The stats, however, told a different story: in 2012, Opta identified Baines as the player who created the most chances in all of Europe's top leagues. His crosses, which were 38% accurate, led to a goal-scoring opportunity every 21.6 minutes, figures that shamed better-known playmakers such as Manchester City's David Silva and Arsenal's Santi Cazorla. Before long, Baines was first choice for the national team and a transfer target for Manchester United (of course, though perhaps he was simply playing better and the data per se had nothing to do with it).

With such an impressive record over the years, it's hardly surprising that Moyes wanted to recreate the structure at Manchester United. Everton, meanwhile, installed Wigan Athletic's Roberto Martínez as their new manager. Martínez had his own reputation for performing above expectations: Wigan had been favourites for relegation from the Premier League every year since they were promoted in 2005; the club consistently had the lowest turnover and attendances in the top flight; their training ground was a converted working-men's club. Somehow they survived – until last May anyway, though they had the consolation of defeating Manchester City to win the FA Cup.

Much of Wigan's resilience was put down to their progressive, young manager. Martínez was known for being obsessive about tactics. The Numbers Game, a recent book that examines the "datafication" of football, noted that he installed a 60-inch pen-touch TV screen at his home and hooked it up with player-tracking software from the performance analysts Prozone. He would watch matches, especially defeats, up to 10 times in order to make sense of what had happened. His response was often unusual and creative: while most teams favour the standard 4-4-2 formation, Wigan under Martínez would shuffle between 4-3-3 or 3-4-3 or 4-2-3-1. In short, he seemed like the perfect fit for a forward-thinking club like Everton.

I meet Martínez at Finch Farm, Everton's training ground on the outskirts of Liverpool. The facility is typically described as "state of the art", but it is still a place where a tea lady will come round to offer you a cuppa and probably a biscuit if you ask politely, too. Martínez is flanked by two of his scouting team, Kevin Reeves and Steve Brown, and we all sit in Reeves's office. There's an iMac on the desk but it is devoid of personal effects and whiffs of fresh paint – it turns out the room used to belong to James Smith, until he moved to Manchester United, and Reeves is just settling in. Reeves was once the most expensive player in Britain – "the first £1.25 million man" back in 1980, he proudly notes – and he has followed Martínez from Wigan.

They have just come in from training. How much data do they collect in preparation for matches? "Every step on a football pitch is measured now," says Martínez, in his unique Spanish-Lancastrian lilt. "We monitor each session with GPS and heart-rate profiles. From a physical point of view, the most significant stats are probably the number of sprints, the sprint distance and the amount of high-intensity efforts a player gets through. We look at these through the season and they give us a good indication of how fatigued a player is and the recovery he needs."

At Everton, each player is tracked in terms of four "corners": technical, tactical, physical and psychological. Data is crucial for assessing the first three categories. On a very basic level, a company such as Opta or Prozone provides multi-camera footage of a player's actions during a match and coaches critique his performance: perhaps they would like him to play more short passes, or – a signature of Martínez's teams – retain possession more assiduously. Detailed feedback will start in some clubs from the under-nines upwards. "You've got so many facilities to look at an individual's performances and you can single out one aspect of his play and measure it – that's significant," he says. "That's unbelievable." Meanwhile, a pair of analysts will be preparing dossiers on the Everton first team's forthcoming fixtures: watching half a dozen of their opponent's previous matches and combining these findings with existing data from Prozone. On the recruitment side, Reeves and Brown liaise with 10 scouts across Europe, who work exclusively for Everton, and keep an eye on the ProScout7 database, which has profiles on almost 130,000 players in more than 130 countries.

Martínez is just as bright and convivial as everyone tells you he is, but he can't hide his deep ambivalence towards, say, ball-retention percentages or the number of successful passes into the opposition's penalty box. Or, to put it another way: he thinks most statistics are useless. "There's a big danger of getting inundated with data and letting it affect your play," he says. "Remember: a player can have 10 shots and all of them are on target but he doesn't score a goal. Or he can have 10 shots and nine of them are off target, but then the last one goes in the top corner. So which stat do you prefer?"

Martínez is not the first to make this point and, in one sense, he is making a distinction between "stats" and "metrics": statistics, on their own, are often meaningless, but through systematic analysis, they can become metrics, which might offer a more revealing measure of a player or a team's performance. Still, it is a surprise to hear Martínez taking this line. Aged 40, with a postgraduate diploma in business and marketing from Manchester University – attained while he was a player at Wigan – you might expect him to be a passionate advocate for analytics. The Numbers Game describes Martínez as a "hero" and its authors, Chris Anderson and David Sally, devote a chapter to his work as Wigan manager, which they approvingly call "Guerrilla Football".

The Everton manager is especially scathing of using data to identify transfer targets – the Moneyball dream of unearthing players whose utility might not always be immediately obvious. There is the famous story of Arsène Wenger signing Mathieu Flamini (the first time) partly due to a statistic that showed he ran 14km a match. Or Liverpool, under their then-director of football Damien Comolli, who spent heavily in 2011 to acquire Jordan Henderson and Stewart Downing, ostensibly because their "final-third regain" percentages – how often they recovered possession in the opponent's penalty box – were so high.

Martínez, and his chief scouts Reeves and Brown, find the suggestion that they would buy a player because of their numbers pretty funny. "You need to see a player and fall in love with a player," says Martínez. "When you see a player, you'll watch his warm-up, the way he speaks to the referee, the way he speaks to other team-mates after missing a chance, the way he celebrates a goal, the way his team-mates react when he scores. Data might help you narrow the margin of error, but the decision is still a feeling. It's a gut instinct."

It is the psychology of a player that Martínez believes is the most crucial aspect of whether a player flourishes or wanes. And it is here that statistics or metrics are most restricted and unreliable. Everton will always scan news reports on a prospective signing and speak to their contacts for character references – some clubs will trawl through a player's Twitter feed and Facebook page – but ultimately the final decision is always an informed gamble. How will a player respond to taking a penalty in the 93rd minute of a Merseyside derby in front of the Kop at Anfield? What happens when your new foreign superstar arrives and struggles to learn English and his wife wants to go home? "Football players are football players once a week," warns Martínez. "The rest of the time they are human beings and fathers and husbands – data doesn't give you that."

While no one contends that the use of data in football will ever be flawless, it certainly continues to become more astute and ambitious. The father of the movement is wing commander Charles Reep, an accountant in the RAF, who codified his first match in March 1950. He would eventually detail and analyse 2,200 games until the mid-1990s, spending around 80 hours on a single match, sometimes writing on rolls of wallpaper. Another pioneer was Valeriy Lobanovskyi, celebrated coach of Dynamo Kyiv and the USSR from the 1970s through to 2002, who spotted the potential of computers to change football when processors were still the size of the team bus. Known for his fastidious match preparations and scientific scouting, he said: "A team that commits errors in no more than 15% to 18% of its actions is unbeatable."

The work of Reep and Lobanovskyi inspired a man you might not expect: Sam Allardyce, now manager of West Ham United. As a player, Allardyce spent the 1983 season with the Tampa Bay Rowdies in Florida; he made only 11 appearances, but the team shared its training facilities with the Tampa Bay Buccaneers NFL squad and he was intrigued by their preparations and that sport's infatuation with statistics. When he became a manager in the early 1990s, he wondered if he might introduce a similar model, but first he had to wait for the technology to catch up with him.

Opta was the creation of a group of management consultants; their first clients in 1996 for their football statistics were Sky Sports and – take a bow – the Observer. Soon they were joined in the market by Prozone, a company that began life as a purveyor of massage armchairs. "Those black chairs you see in motorway service stations that you put £1 in," says Paul Boanas, Prozone's senior account manager. Early interest in Prozone came from another unlikely innovator, Steve McLaren, then a coach at Derby County. He liked the chairs, but the players got bored sitting in them for 15 minutes after every training session. He asked: "Couldn't they watch videos of the game while they're doing it?"

McLaren, who would move on to coach Manchester United and then manage England, and Allardyce, who by this time was manager of Bolton Wanderers, would become Prozone's earliest and most devoted customers. For Big Sam in particular, the new software was addictive: he hired a team of young sports-science graduates and used the video analysis to mould Bolton's style of play. They calculated that any team that ran further and faster than their opponents would win or draw 80% of their matches. Their players relentlessly practised throw-ins, corners and free-kicks – targeting "pomos" or positions of maximum opportunity – and scored around half their goals, far above the league average, from these set-pieces. Allardyce stitched together a team of misfits, old-timers and foreign mercenaries, led by Gary Speed. When he arrived on a free transfer in 2004, Speed was 35, but his stats – 12km a game, a pass-completion average of above 80% – suggested he could still be useful. He became a talisman for Bolton for the next four seasons.

Big Sam's Bolton defied logic: they finished in the top eight of the Premier League every season between 2003 and 2007, and twice qualified for the Uefa Cup. But "pomos" did not enter the lexicon of the data revolution and many of his ideas now seem outdated.

Allardyce remains committed to metrics, but his greatest contribution to the movement might just be the people he inspired. Bolton alumni now head the analytics departments of the most ambitious clubs in world football: Ed Sulley is head of performance analysis at Manchester City, while Gavin Fleig is City's head of technical scouting; Dave Fallows is head of recruitment at Liverpool. These men could be just as influential in shaping the future of their clubs as the managers, Manuel Pellegrini and Brendan Rodgers.

There is a clear shift of power taking place at some clubs, and the use of data analytics is at the heart of it. At a time when the average tenure of a Premier League manager is just over one year – seven have already been sacked this season – the idea of entrusting all elements of player recruitment and long-term strategy to the manager is anachronistic. That certainly seems to have been the conclusion of the owners at Manchester City and Liverpool, as well as a club such as West Bromwich Albion, which shares power between the manager and a director of football, or sporting and technical director as they now call the position.

"The perfect model in the club's eyes is to have everything set up and just drop in the manager and he's only allowed to bring two members of staff with him – that's what clubs would like," says Prozone's Boanas. "When the average lifespan of a manager is so short, they're going to think, 'Why would I plan for the future, when I might be gone in six months? Bollocks to that!' Instead of signing a young player, they're going to bring in a 31-year-old who's got a proven record, who they've worked with before. It's a very short-term view."

Chris Anderson, author of The Numbers Game and a political scientist at New York's Cornell University agrees. "Incentives are incredibly important," he says. "The right incentives in my mind are the ones that keep this club healthy beyond next Saturday and perhaps beyond this month and even beyond this season. The place where a manager has a long tenure – like David Moyes at Everton and Arsène Wenger at Arsenal – that person's incentives for themselves and for the club are reasonably closely aligned. But, the world we live in, sometimes that person isn't the manager."

At a certain point, however, Allardyce's Bolton protégés, the men now driving the use of data analytics in British football, hit a wall: they were sports scientists, not mathematicians. This frustration was eloquently expressed by James Smith, then still at Everton, at the Elite Minds in Sports Analytics Summit held at Arsenal's Emirates stadium last November. It can be a lonely business being a quant in a football club, and the three-day seminar – with presentations by everyone from YouTube to the performance director of British Bobsleigh – fell somewhere between a show-and-tell and a self-help meeting.

"At Everton at the moment we're still very much in a world of GCSE maths," Smith said. Cue an intake of breath in the room, and much frenzied tapping on laptops. "We look at averages, we look at benchmarking, we are in the world of bar charts. At the moment we are not doing more sophisticated regression analysis" – a statistical process used for predicting future outcomes – "but we know that is probably the way forward and that's where we hope to be before too long. But at the moment that tends to be the bigger clubs, the better-resourced clubs really."

Smith contrasts football unfavourably with American sports, notably baseball and NFL. "Typically the guy dealing with the data in an English football club at the moment is a sports science graduate – which I am," he said. "But very often in America you might have somebody who went to Harvard and did a law degree then did a computer science masters at MIT. One of the issues in English football is we don't spend enough on staff: quality or quantity. And that's partly because we spend so much money on transfer fees, player salaries, agents' fees that there's not enough left. It's crazy."

There are, in fact, some whip-smart mathematicians working in English football, but, because of the traditional approach of most clubs, they are more likely to be employed by a betting company or a data generator such as Prozone. In an attempt to address this disparity, a fascinating initiative was launched by Manchester City's Gavin Fleig in August 2012. Called MCFC Analytics, the club released a large archive of data collected by Opta from the 2011/12 season. It was an "open source" call to arms for bloggers, PhD academics, anyone with an inquisitive mind and an interest in football who wanted to mess around with numbers.

The inspiration for the experiment was baseball, specifically Bill James, a janitor whose after-hours statistical analysis revolutionised that sport. "I want our industry to find a Bill James," Fleig told Simon Kuper. "Bill James needs data, and whoever the Bill James of football is, he doesn't have data because it costs money."

MCFC Analytics ended after a year and it's hard to determine if it was a success or not. The interest was certainly there – more than 1,500 users accessed the information in the first 36 hours – but there was criticism of the "basic" dataset that was released. Dr Howard Hamilton, chief executive of an Atlanta-based consultancy firm Soccermetrics Research, who holds a PhD in aeronautics and astronautics from Stanford University, described it in a blog as "woefully inadequate".

"It wasn't our deepest dataset by any means, but it was relatively deep ," says John Coulson, head of professional football services at Opta. Nevertheless, Coulson can't see the experiment being repeated in the near future: "It was a one-off thing: 'Here's something to have a go at, get your teeth stuck in.' But it's not sustainable for us as a business to release all of that data every year."

Football clubs are intensely secretive about the specifics of their use of data, especially where they believe they might have a competitive advantage. So I ask Marcus du Sautoy, professor of mathematics at Oxford University and keen Arsenal fan, what impact a greater numerical literacy could have on the game. "Football is much more of a game of chess than people realise," he replies. "It isn't random what each team does from one game to the next. There are patterns. And the strength of mathematics is to change an activity into numbers and to spot patterns and predict things into the future. That's essentially what the hedge-fund guys are doing."

Du Sautoy believes we should look at the football pitch as a network, with channels connecting the 11 players – "It's like a mini-internet!" he exclaims. A successful team – Barcelona are the perfect example – has a special ability for keeping these connections open, but there's no reason why all teams could not analyse the dynamics in a more theoretical way. Du Sautoy also thinks that coaches would benefit from a greater willingness to think outside the box, so to speak. He uses the example of a free kick: why does the defending team always line up with a wall in front of the kicker? Perhaps that is the most effective way of blocking the ball, but they could test the hypothesis more methodically.

"Football is incredibly conservative," says du Sautoy. "Having people who come from a different mindset could actually give a team like Arsenal or Liverpool a real edge." Then, at least half-seriously, he ventures: "If Wenger wants a mathematician on the bench at Arsenal, I'd be very happy to come along."

It is easy to become carried away with the possibilities of data analytics. At the Elite Minds in Sports Analytics Summit, another speaker was Brian Prestidge, head of analytic development at Bolton Wanderers. He revealed that, since their goalkeeper had started studying the stats on the opposing team's penalty taker, he was actually saving fewer penalties (just 9% in the last two seasons). "We took away the human element, the player's instinct," said Prestidge. "But that's not to say there are no advantages in analysis."

If data is to have a greater influence in how football teams are run, it is likely to be at the instigation of the club owners – such as Liverpool's John W Henry, who made his fortune on the stock market and whose other team is the Moneyball-inspired Boston Red Sox – rather than the managers. Players, too, might also demand it: at the Elite Minds summit, Ben Smith, head of development performance systems at Chelsea, explained that young players – such as Eden Hazard – had grown up with data and constant feedback and now expect it after every match and training session; this contrasted with the older generation who can often be more entrenched.

Of course, a manager will never admit that a number-cruncher might do his job just as well – or, heretically, even better – than he can. "And if a manager is doing something sophisticated or analytical, he won't want to advertise that to the world," says Anderson. "It makes them look less good and it makes them look geeky, too. In this manly world of football, you don't want to be known as a pinhead. That's the worst of everything!"

Anderson recently floated the idea that a Premier League club could reduce its squad from 25 players to 24, and use the savings to employ a handful of maths graduates, who would doubtless earn less in a year than some players are paid for a week. No one seriously expects any club to take up the suggestion. At Finch Farm, I ask Martínez if he is envious of Manchester City's 11 analysts, working behind the scenes to plot their next opponent's downfall.

He shakes his head. "I don't start with 100 people and say, 'How are they going to help me win a football game?' Doesn't matter if you have 100 or 3,000 people. It can dilute the quality. We are in a position where we've got enough to do everything we want. I don't think we feel frustrated or we need to get more finances. No, I think we are very much efficient."

Football is a game of passion, and part of every fan would die if the game were reduced to a soulless set of calculations. But equally, any club or manager that denies the power of data are placing themselves at an enormous disadvantage. In one sense, this could be a positive development: football has historically been dominated by the teams with the fattest wallets; in the age of analytics, clubs should be rewarded for innovating and there is a greater motivation for cash-strapped teams to lead the way. Brains can trump financial brawn. Though, it should be noted that right now Manchester City are leading the field in both categories.

Sitting in the stands, fans will likely stay, at least partially, in the dark. When a substitute comes on and scores with his first touch, do you credit the genius of the manager or the calculations of his performance analysts? In the moment – particularly if you're a Manchester United fan – you'll probably be too ecstatic to care.