My first mistake at the Guardian came, agonisingly enough, in my first article as a member of staff, and remains memorialised on the internet to this day. My subject was not a theoretical physician, a note at the end politely explained, but a theoretical physicist. Ten years later, it still makes me shudder.

Still, I felt a little better about it when I saw the run that Apple’s artificial intelligence system went on recently. In its automated summaries of news notifications sent out to some iPhones, it asserted that Luigi Mangione, the man accused of killing the UnitedHealthcare CEO Brian Thompson, had shot himself. It said that Benjamin Netanyahu had been arrested. It said that Pete Hegseth, not then confirmed as Donald Trump’s secretary of defense, had resigned. And it said that Rafael Nadal was gay. For the avoidance of doubt: he didn’t, he wasn’t, he hadn’t and he isn’t.

This litany of howlers were the product of the Apple Intelligence AI system, which was rolled out across the company’s recent phones in the UK in December. There was no way for publishers to opt out, and an inscrutable little arrow next to the source news organisation’s logo was the only marker that this was Apple Intelligence’s output, rather than their own. After the BBC ran a series of richly indignant news stories about the phenomenon, Apple did something it almost never does: it reversed course, and paused the service. (Full disclosure: my wife works for Apple.)

You could see why Apple pulled the plug: the story was funny and weird enough to get traction. Saying, as another of the summaries did, that Luke Littler had won the darts world championship final before he’d actually played in it, is the six-fingered hand of news events: something that almost looks real, but reveals itself as fatally and hilariously flawed if you pay close enough attention. If it had been up and running last summer, you dread to think what the summarising process might have done to the news of the assassination attempt against Donald Trump.

Still, the central dynamic here is of two implacably opposed cultures: one that embraces public iteration, and the many missteps that users will endure along the way; and one which knows that if it keeps getting things wrong, the whole point of its existence will be annulled. There’s no realistic way to reconcile these tendencies, but you might think a journalistic carve-out would be a grownup solution.

We can’t expect Apple Intelligence to shudder when it reflects on these errors, because it doesn’t actually comprehend what a fact is – it just knows what one sounds like. What’s perhaps more alarming is the view from the other end of the telescope: that we might very easily not have known about these mistakes, and that there were almost certainly many others that went entirely unnoticed.

All of this came to light only because journalists – at the BBC, but also ProPublica and the Washington Post – happened to receive the alerts in question. The summaries were locally generated on users’ phones, and Apple has no way to track them. That means that it is very likely that other users will have received inaccurate alerts, and blamed them on journalistic incompetence rather than a bot gone wild – or quite reasonably believed them to be true. There are probably still iPhone users roaming the streets under the impression that Father Christmas gave the king’s speech, or whatever.

The BBC wasn’t the only news organisation to notice the problem, but it was the only one to really go after it publicly. The vigour of its coverage was partly a submerged message to readers. “It didn’t hurt that it let people know that if they were seeing this, it wasn’t the BBC doing it,” one journalist there says. “But it was also about drawing a line in the sand with Apple and saying – look, this looks like it comes from us, and our reputation is completely founded on being reliable.”

Some might scoff at this, and point out that news organisations make their own mistakes all the time – more consequential than my physicist/physician howler, if less humiliating. But cases of bad journalism are almost always warped representations of the real world, rather than missives from an imaginary one. Crucially, if an outlet gets big things wrong a lot, its reputation will suffer, and its audience are likely to vote with their feet, or other people will publish stories that air the mistake. And all of it will be out in the open.

You may also note that journalists are increasingly likely to use AI in the production of stories – and there is no doubt that it is a phenomenally powerful tool, allowing investigative reporters to find patterns in vast financial datasets that reveal corruption, or analyse satellite imagery for evidence of bombing attacks in areas designated safe for civilians. There is a legitimate debate over the extent of disclosure required in such cases: on the one hand, if the inputs and outputs are being properly vetted, it might be a bit like flagging the use of Excel; on the other, AI is still new enough that readers may expect you to err on the side of caution. Still, the fundamental difference is not in what you’re telling your audience, but what degree of supervision you’re exercising over the machine.

In the end, the most telling part of all this may be Apple’s reaction. Even if it stonewalled for a month after the BBC’s first story, it is striking that a company with a longstanding reputation for defensiveness ultimately relented. But it will undoubtedly bring the product back, and it has not said where the bar is for accuracy that it will consider sufficient.

The really interesting question is why Apple made its use of AI in these summaries so invisible – if only there were two letters representing the technology that might easily be assembled into a legible icon! It’s a small detail, in one sense. But it also points to a much larger problem: when you’re dealing with the introduction of a revolutionary new technology that your users don’t really understand, adhering to your defining mission for simplicity may make things pretty complicated.

The truth is, even if Apple is less thoroughly defined as an AI business than some of its big tech rivals, it is nevertheless a participant in a gold rush. The AI products that stick with us over the next few years may be a bit like the Qwerty keyboard against various alternatives, or Internet Explorer’s victory over Netscape: worse, but ultimately victorious because of their unconquerable scale. If so, a frictionless experience might be a very effective strategy.

If the complaints of even the most venerable news outlets are part of the price, it won’t even touch the sides. In its latest software update last week, the same one that disabled the news summaries, Apple turned on AI by default for the first time.

• Archie Bland is the editor of the Guardian’s First Edition newsletter