Once a month, every month, more than 8,000 strangers pay James Hoffmann a total of £16,263 so he can go out and buy coffee machines. Hoffmann, 41, from London, is an author, business owner, coffee connoisseur and, above all, a YouTuber: more than 900,000 people subscribe to his channel, on which he discusses everything to do with beans and brewing. Around a third of Hoffmann’s videos are product reviews: grinders, espresso machines, storage canisters and filters have all been scrutinised by him.
Hoffmann’s monthly £16,000 comes from Patreon, a membership platform that allows fans to pay creators a regular fee. The money is intended to keep him impartial: it enables him to buy machines to review directly – just like you or me – instead of getting them on loan from brands.
“It is about building trust,” Hoffmann says. “If I get loans or freebies from manufacturers, that changes the nature of the review, because you don’t bite the hand that feeds you.” In his review of a £299 Aldi Ambiano espresso machine in April 2020, Hoffmann didn’t hold back. “This is a stupid coffee machine. I hate this coffee machine,” he said in the video, which has more than 2m views (Hoffmann objected to the poorly aligned stickers, the pump pressure and tricky mechanisms that resulted in spilled coffee grounds).
Hoffmann has a long history with coffee: he worked for an espresso machine company in 2003, and began to enter coffee competitions in 2005. He has since been crowned both UK and World Barista Champion – “part dog show” – in which he had to make a coffee cocktail in front of four judges. His recipe involved a “flavour marriage” of tobacco and almonds. In 2008, he co-founded his own coffee roasting company, and in 2014, published The World Atlas Of Coffee.
Undeniably, Hoffmann is well placed to review coffee machines, but the question is, why are people willing to pay him to do it? We live in a golden age of reviewing – Amazon, Google, and Facebook publish millions of free reviews, while hundreds of expert product testers at the campaigning consumer organisation Which? review more than 8,000 items every year which anyone can read for a £7.99 monthly membership fee.
But all these reviews don’t create clarity for consumers. On the Aldi website, where 48 reviewers have bestowed 4.2 stars on the machine Hoffmann reviewed, BrianH from Sheffield thinks it’s a “great product at a great price” – he’s one of 40 people who rated the machine at four or five stars. Two people gave it two stars and six people gave it just one: Mrs D says it’s faulty, Daniele found it noisy, and Jenny held down the caps lock key to complain “WATER NOZZLE DIDN’T WORK”.
In a world full of confusing consumer reviews – where, during the pandemic, online shopping sales hit a 13-year high – a demand for authenticity is being supplied by social media stars. Think of a household appliance and you will find a corresponding expert on YouTube or Instagram. From vacuum cleaners to prams, fragrances to phones: there’s a reviewer for that.
“Video is humanising,” Hoffmann says via Zoom from his studio, which has two sets (a kitchen and an office) where he can test products and film reviews. “You get to see someone use the product, to see it in situ, in use – all of those things are very different from Amazon reviews.” When Hoffman’s viewers – mostly male, aged 20-45 – watch a video, they want to be entertained as well as educated. Some viewers have described his style as “gentle chaos” – he has a widow’s peak of expertly swooped silver hair, tortoiseshell spectacles, a penchant for woolly jumpers, and a soft voice. But he’s not afraid to strain doughnuts into coffee or brew grounds in a bripe – a coffee pipe.
And why not seek out a friendly face when, according to a 2020 survey by marketing company BrightLocal, 80% of consumers believe they have read a fake review in the last year? The same company estimated in 2018 that 61% of electronics reviews on Amazon are fakes solicited by sellers and manufacturers.
But not all niche reviewers are created equal. While some take great pains (and pounds) to remain impartial, others are paid directly by brands to promote products. Some rigorously test items for weeks, while others trade in first impressions. Are YouTubers an oasis in the wild west of internet reviewing – or are they part of the problem? How do you work out who to trust?
***
There’s a cabin at the bottom of Matthew Lock’s dad’s garden containing row after row of vacuum cleaners. The Liverpool shed doubles as a workshop; this is where Lock, 16, tests vacuum cleaners on squares of carpet and tile on which he has scattered flour, oats and other dry household debris. Sometimes, he will weigh a dirty square of carpet before and after he vacuums to determine exactly how much the machine managed to pick up. He even has a smoke machine to test filtration systems.
“There was no one who really explained the insides and outs of a machine in a short, concise way that cut through the marketing spin and got to the facts,” Lock says. This was why he started his YouTube channel in 2017, when he was 12. Today, he has 3,500 subscribers who are 91% female and mostly aged 18-30. On his Instagram page – The Vac Mat Product Reviews™ – he has an additional 28,700 followers.
Lock speaks with the authoritative, matter-of-fact tone of someone three times his age. He developed an interest in floor-cleaning appliances when he became enchanted by Henry the vacuum’s face, aged three. When he was five, he woke up one morning and unscrewed the radiator cover from the wall, “just to see how the radiator worked behind it”. At 11, after a family friend brought him a broken vacuum cleaner to tinker with, he started a repair business and was “swamped” with work. That motivated him to begin posting reviews: “I started seeing that people were buying the wrong product for their needs.”
Initially, Lock’s friends were unimpressed. “At first people thought it was a bit mad,” he says, “Most kids were interested in football and those sorts of things.” Thanks to his impressive following, however, Lock’s peers have been sucked in. He is now at college (studying physics, engineering, business and maths) and replies to dozens of messages daily from viewers seeking advice. When he’s older, he hopes to work in research and development. For now, he runs his tests.
Before taking a machine to his shed, Lock uses it around the house for a couple of weeks. “On Instagram, you’ll see people vacuum a rug or something with a 20-year-old Dyson, and then they come along with a brand new cleaner out of the box… They’ll say, ‘Oh! Look at all this that it’s picked up!’ But you’re comparing an apple and an orange there, aren’t you?”
***
Safwan AhmedMia, from Leicester, also prides himself on his rigorous tests. On his channel SuperSaf ( 1.75 million subscribers) AhmedMia reviews phones, cameras, laptops, tablets, speakers and headphones. He also does smartphone camera comparisons which “a lot of the audience come to my channel for”. AhmedMia’s audience is 90% male, mostly aged 18-35. (AhmedMia prefers not to reveal his own age). It takes him three days to create a video comparing two cameras – he visits multiple locations to see how they perform in different lights, tests the video capabilities and (at the behest of his audience) takes a number of selfies to see which is the most flattering piece of tech.
To eliminate extraneous variables when testing, AhmedMia has built his own smartphone rig – two smartphone mounts screwed into a metal plate with a handle attached, a sort of mutant double-selfie stick. He started reviewing because he has “always been tech support for the family” – grandparents, aunts and uncles would call him before buying new kit. This means that beyond technical specifications, his reviews focus on ease of use and value for money, and the tone is conversational, “like I’m talking to a friend”. This can have drawbacks – an avid fan once turned up at his house.
Fans might love AhmedMia and Lock, but how do their self-taught product testing skills compare to those of Which? – the largest not-for-profit consumer organisation in the UK, with testing labs across Europe and 64 years of reviewing to its name? A Which? endorsement is coveted by companies, but is it valued by consumers? If we review the reviewers, who comes out top, and who leaves whom behind in the (carefully scattered, sucked and weighed) dust?
In the Which? laboratory, bags of pet hair are stored in the freezer. “Pet hair degrades over time,” Matthew Knight – principal researcher at the organisation – explains, so the freezer helps to preserve it for vacuum cleaner suction tests. In 2013, Knight joined Which? as its “vacuum cleaners, boilers and lightbulbs expert”, though he now spends his time identifying new product categories to review (he recently spent a day riding e-scooters down a hill while wearing a backpack so he weighed the same as the average UK man).
Which? examine most products in a lab setting, and Knight estimates that each product undergoes 70-100 different tests. “Our golden rules are that the tests are robust, repeatable, rigorous, and relevant – the four Rs,” he explains. Each test is run at least three times to ensure there are no anomalous results. At the moment, Knight is testing salt and pepper grinders by scrutinising the uniformity of the grounds. Five separate testers grade the consistency of the grains by eye – Knight says there’s no hard and fast rule about how many testers work on a product, but normally at least five people work on “subjective” tests like this.
Not all Which? testers are human. For new phone tests, there is a robot arm that flicks through the apps, contacts, map and camera to see how long the battery takes to run flat. The team then runs subjective tests. Every assessment is weighted differently – for example, the quality of the photographs will affect the overall score more than how easy it is to navigate the menus. Any phone that scores less than 45% automatically becomes a “don’t buy” product.
Like Lock, a Which? vacuum tester will try a machine on spilt rice and cereal, but they also put it in a sound chamber and measure the noise it makes with decibel meters. A former nuclear physicist uses a “fine particle machine” (“some fancy piece of equipment which I don’t really understand,” Knight says) to count “the tiny, tiny microns” that manage to pass through a vacuum’s filter. “People have allergies, they don’t want those particles blown back into their face or around the room as they have been sucked up,” he says.
Lock once bested Which?. In October 2019, he watched a Which? video comparing Gtech and Dyson vacuums, and was surprised to see the Which? experts complain that the Gtech wasn’t picking up dirt properly. Lock looked closer and realised the team had assembled the machine incorrectly (it was a social media video, not the official lab test). Lock messaged Which? to say the brush roll on the machine was unclipped and not rotating properly. In response, Which? reviewed the clip and took the video down.
Knight almost seems like an older version of Lock – they both have fair hair and a wealth of knowledge, not to mention a similar taste in polo shirts. Yet Knight’s personality doesn’t come into his job – after tests, he’ll analyse the results spreadsheet and write a review. It is the Which? brand – not Matthew Knight – that consumers have learned to trust. At present, Which? has more than 600,000 paying members and the company says it supports more than 6m purchasing decisions annually.
“We have the freedom to tell people if products are rubbish, which we often do with a smile on our faces,” Knight says. “You won’t find many negative reviews on YouTube or by lots of our competitors.” Knight says that because Which? never receives free products, it remains completely impartial. “We’re not in the pockets of any manufacturers.”
Around 80% of the products AhmedMia tests are loaned to him by companies, but he says this does not affect the content of his reviews: most newspapers and magazines also accept products on loan. (The Guardian’s policy on consumer technology reviews is to buy devices outright or to ensure that borrowed items are returned to the manufacturer or donated to charity after the test period.) Lock buys machines and also accepts them from brands, but says he isn’t afraid to be critical of freebies.
Hoffmann’s followers pay for his products, but there is a man with a pram – or 100 – who goes even further. Wade Elliott, 41, is a Californian living in Oslo who runs the Stroller Workshop, a pram review YouTube channel with just over 16,000 subscribers (60% women, mostly new mothers). Like Hoffmann, he earns money via Patreon so he can buy products – he makes £500 a month from 189 people. Elliott believes online buggy reviews are often biased. “Manufacturers have it kind of locked down. They have outreach programmes for YouTubers and bloggers – you sign a contract with them. So that’s how most of the channels will work, they get the free product by signing a contract.”
Elliott shuns contracts and also won’t use affiliate links, which allow a creator to earn money every time they are clicked (the vast majority of big social media personalities use affiliate links). “If you are making your money through affiliate links, then you’re only going to make money if people buy what you review. So if you give it a bad review, nobody’s going to click on that affiliate link,” he says, adding that he has been “chewed out” by brands whose products he has slated.
Elliott says he got into prams by a fluke. He was finishing an MA in philosophy when his son was born and “poor and still living in student housing”, he got a secondhand buggy that “wasn’t all that great”. Elliott fixed it up and sold it on for a small profit. Within a few months, he had built a stroller repair business, which grew into online review.
Elliott learned about prams and strollers on the job. Via Patreon, he is able to offer fans one-to-one feedback on their issues, and his YouTube comment section paints a clear picture of the impact of his findings. “Seeing these unbiased, not profit- or trend-driven reviews coupled with a lot of technical expertise has really helped me made up my mind,” a commenter named Stefanie wrote a year ago. Her post is nestled between hundreds of people describing their babies’ (and husbands’) heights and weights, clamouring for Elliott’s expertise.
***
In April, I bought my mum a dehumidifier. Looking at my internet history now, I see that I spent more than two hours over the course of two days clicking between Amazon, Argos and Which? (then clicking off Which? because I wasn’t a member). I quickly became overwhelmed.
In the end, I settled for a £150 model on Amazon that had the most realistic-sounding reviews. While other models had higher ratings, the feedback was vague or repetitive, or otherwise suspicious. Later that month, the Competitions and Markets Authority (CMA), the UK regulator, prompted Facebook to delete 16,000 groups in which users facilitated fake feedback by offering money in exchange for product reviews. Facebook said it would suspend or ban users who “repeatedly” create groups dedicated to fake reviews, and introduce new technology to flag such groups automatically. The CMA is currently investigating whether Amazon and Google are taking sufficient action to protect consumers. Amazon has said it removed more than 200m suspected fake reviews globally before they were seen by customers in 2020. Google said the company has strict policies on fake reviews and has disabled user accounts over breaches.
Which? has also undertaken a number of investigations into fake and incentivised reviews. In May 2020, the organisation ran a shopping task with 10,000 participants and found that “fake reviews make consumers more than twice as likely to be misled into choosing poor-quality products”. In June 2021, it found that bestselling Amazon products show evidence of manipulated reviews, where reviewers were offered vouchers, freebies or money to post a five-star review or remove a negative one.
Is it any wonder that people are turning to niche reviewers? Dr Frederik Situmeang is a marketing lecturer at the University of Amsterdam. He believes reviews by social media stars are “very influential and will become even more so in future”. “We all live now in the digital age, which allows us to access enormous amounts of information and we definitely do not have the time to process all this information,” Situmeang says. “Social media influencers are relatable to us. We see them as consumers, just like us.”
Sometimes, then, people turn to niche reviewers not because they think they are the most objective or rigorous, but because they like them. Monika Cioch, 26, lives in Birmingham and reviews fragrance for more than 61,000 YouTube subscribers. In short, attention-grabbing videos, she tells you which scents are the sexiest or most summery, or most likely to get you compliments.
Cioch has always loved fragrance. She looked into enrolling at perfume school in France, but couldn’t afford it. Instead, she set up a YouTube channel. To review a new fragrance, Cioch will judge its scent but also its value for money. “The channel is based on my subjective opinion about fragrances, so I have to always make sure I stay true to my taste, so my audience can rely on that,” Cioch says. Viewers know she likes “timeless, classy scents that are not too harsh”. She also warns her audience how to spot fakes (the longevity of the scent is one clue, and she compares packaging and logos).
Cioch doesn’t earn enough to live off her reviews. She has grown an audience which is 90% male. “Men like that they were hearing about scents from a woman’s perspective, so I decided to go in that direction,” she says.
Can niche online reviewers be totally altruistic? Many social media stars are paid directly by companies to wax lyrical about their products – while paid promotions have to be marked with the hashtags #ad and #spon, a paid endorsement can be confusing for those who visit a creator for impartial reviews.
“I have seen other influencers sell their soul to make a quick buck,” Lock, the vacuum enthusiast, says. “I watch reviews basically shouting at the screen. At the end of the day when they go, ‘This review was sponsored by the opposite manufacturer’, you go, ‘Ey up. That’s not right’.”
Lock has been paid to promote products, such as a tool that picks tangled hair out of vacuum cleaner heads, but he has never taken money from a machine manufacturer. He says his desire to remain credible means he makes no profit from his videos, which are “more of a public service than a source of income”. There are perks, though. Lock has such a good relationship with Dyson that he has been invited to film inside the company’s research and development facilities, despite Dyson’s strict no-filming policy. “I do get around,” he says with a hint of pride.
Each YouTuber chooses how to navigate the paid partnership landscape differently. AhmedMia does “a lot” of brand partnerships but never offers sponsored reviews. “Any reviews have to be impartial. I don’t offer sponsored comparisons or anything like that,” he says – generally, his adverts are integrated as asides into other videos, or he might do a sponsored unboxing of a product. “All content is marked whenever it’s sponsored,” he says, adding that he rejects 90% of paid offers he receives, “because it doesn’t fit my level of quality and [is not] something that I would be comfortable promoting”. He has built audience trust over time by being “honest and open”.
Similarly, Cioch has always had a clear vision of which sponsorships she will accept. “My general rule is that I have to stand for what I am promoting,” she says, “I have to like the fragrance, the brand, or use it myself. My reputation is more important than any sponsorship.”
Hoffmann doesn’t take sponsorships or free products, but admits he can only do this because he has other sources of income. “I’m very fortunate that I can make that decision without the pressures of, can I pay my rent? Can I eat? If you’re a creator, financial security is difficult.”
Combined, the five YouTube reviewers interviewed here have around 2,730,500 subscribers – about 4.5 times the number of people who pay to join Which?. It’s an extraordinary audience, but with every creator making up their own rules, there remains a lack of clarity for consumers. I don’t think I would buy the Aldi coffee machine and know where I would go to find out about prams, but I’m not sure where I will turn the next time I need to make a big purchase. How can you truly scrutinise a product when you only see it online? Perhaps I should just invest in a camera, email some companies, and become a YouTube reviewer myself.
