Researchers handed people a set of genuinely hard questions, and gave them a way out: they could always decline to answer. Without an AI assistant available, participants took that option often. With one available, they almost stopped using it entirely, and their accuracy fell to roughly a third of what it had been while their confidence nearly doubled.

That last part is the finding worth sitting with. Not that people got worse. That they got worse and felt better about it.

The study is a preprint from Valerio Capraro, Chiara Marcoccia, and Walter Quattrociocchi, published July 15, covering five experiments with 3,132 participants, four of them preregistered and one a direct replication. That's a larger and more carefully structured piece of work than most of what gets written up as an AI study, and the design choice at the center of it is what makes the result matter.

They rigged the questions so the AI would be wrong

This is the part that separates the study from the usual "AI gets things wrong" coverage. The researchers deliberately engineered questions where the model would fail, using things like visual details from films that a language model has no reliable way to know. They used a model that was consistently wrong on those questions on purpose.

That design isolates the behavior from the accuracy. If people had gotten worse because they were leaning on a tool that happened to be unreliable, the fix would be obvious: use a better tool. But that isn't what the researchers were measuring. They were measuring whether the presence of an answer changes how willing a person is to admit they don't have one. It does. Some participants who would have answered correctly on their own consulted the AI and became wrong.

Valerio Capraro put the stakes plainly in an interview with The Register: the capacity to say "I don't know" represents the recognition of the limits of your own knowledge. Strip that out and you haven't just lost an answer. You've lost the signal that tells you to go find one.

The detail that should worry business owners

Here is the finding that turns this from an interesting result into an operational problem: it held whether the advice was actively requested or simply displayed.

Think about where that puts you. Your team may have a sensible policy about when to use AI tools. That policy assumes using AI is a decision someone makes. Increasingly it isn't. Summaries now appear at the top of search results before anyone asks for one. They appear in email clients, in help desk software, in CRM records, in accounting platforms, in the sidebar of tools your staff opened to do something else entirely. Nobody chose to consult a model. A model answered anyway.

The researchers also tested whether money fixes it. Paying participants for accuracy and penalizing them for errors did help. It moved willingness to suspend judgment and accuracy in the right direction, and people sought and followed the AI's advice less. But both measures stayed far below the no-AI baseline. If a direct financial incentive to be right only partly counteracts the effect, a paragraph in your employee handbook is not going to do it either.

That is why this reads to us as a design problem rather than a discipline problem. You cannot train your way out of an interface that supplies a confident answer to every question before anyone has decided a question needs answering.

Where the deleted pause actually costs you money

The wrong answer is rarely the expensive part. The expensive part is the thing that would have happened instead.

Quoting work. Someone estimates a job from a plausible-sounding summary rather than pulling the last three comparable invoices. The number is defensible and wrong, and you find out at the end of the project.

Diagnosing a customer problem. Support gives a confident explanation instead of saying "let me check the account and call you back." The customer accepts it, the real issue persists, and the second contact is angrier than the first.

Reading an agreement. A renewal clause gets summarized rather than read. The summary is fluent. The clause has a carve-out the summary flattened.

Checking whether you can do the thing. An inquiry comes in about work slightly outside what you normally handle. The confident answer is yes. The correct answer was "I don't know, let me look at it."

In each of those, an employee who said "I'm not sure" would have triggered a small, cheap, useful action: open the file, call the supplier, ask the person who knows. That pause is one of the most valuable things in an operating business, and it's almost invisible until it stops happening. We've written before about which parts of a workflow are safe to automate and which quietly hold everything else together, and this is the same lesson arriving from a different direction. The step that looks like hesitation is often the step doing the work.

Build the "I don't know" back in

None of this is an argument against using AI in your business. We build AI development work into client systems regularly, and we think most companies are still underusing it in the places where it genuinely helps. The argument is about what a good implementation looks like.

A system worth trusting can decline. When the retrieved records don't support an answer, it says so rather than producing a fluent guess. It shows what it drew from, so a person can check the source in one click instead of taking the summary on faith. It routes low-confidence cases to a human instead of resolving them silently. And it's scoped to a domain where you actually have the data to be right, rather than pointed at the whole world and asked to be helpful.

That is harder to build than a chat box, which is exactly why so much of what gets shipped is a chat box. Wiring a model into a business process properly means knowing your data, your edge cases, and the specific decisions where being confidently wrong is expensive. It's the difference between a demo and something you'd let touch a customer record. Most of the useful work in custom software right now lives in that gap, and it starts with the same question we ask at the beginning of every automation conversation: what should this actually be doing, and what should it refuse to do?

The study's authors put it in terms of a metacognitive threshold, the internal line where a person decides they know enough to answer. AI moved that line. In your business, that line is a control. It's the thing that sends someone to check before they commit you to a number, a promise, or a contract.

Worth knowing where yours currently sits. If you're putting AI into a process where somebody's judgment used to be the safety net, that's a conversation we're glad to have before it ships rather than after.