A couple weeks ago I wrote about NPC mode: the moment you stop thinking and start going through the motions. A Wharton study that’s been making the rounds puts a number on it.
Shaw and Nave gave 1,372 people logic problems with an AI assistant that was secretly right half the time and confidently wrong the other half. When people used the AI and it was wrong, they followed its bad advice 80% of the time. Their accuracy dropped 15 points below what they’d have scored with no AI at all. And their confidence went up.
The researchers call this “cognitive surrender.” The term is new. The behavior isn’t. Anyone who’s moved forward with an AI recommendation because questioning it felt like more work than trusting it has been there. The paper’s contribution isn’t discovering the phenomenon. It’s measuring how strong the pull actually is: even when participants were paid for accuracy and given immediate feedback on every answer, the surrender pattern persisted. It shrank, but it didn’t go away.
Here’s what caught my attention most. Their Study 3 (with incentives and per-item feedback) more than doubled the rate at which people rejected bad AI advice. Not because they were told to be more careful, but because they had a tight feedback loop with real stakes.
The paper’s recommendations point toward design-level solutions: uncertainty indicators, digital literacy, adaptive nudging. That’s valuable framing. But for practitioners, there’s a more concrete version of the same insight: build verification into the workflow so it happens whether you feel like thinking critically or not. Don’t rely on willpower to resist a confident, fluent answer. Engineer the checkpoint.
As a software developer, I recognize this pattern every time I review a pull request (someone else’s proposed code change). It’s easy to see clean-looking generated code, skim the explanation, and click approve. The drift from “I evaluated this” to “this looks fine” is subtle and fast. The antidote isn’t vigilance. It’s structure: a forcing function that makes you articulate what you checked and why before you sign off.
That principle applies anywhere AI output gets folded into a decision: a financial analysis, a legal summary, a marketing brief. Structure beats willpower.
I wrote Spanning Change around exactly this idea: not “be more careful with AI” but specific, repeatable practices that create the feedback loops cognitive surrender doesn’t survive.
The study is a preprint from Wharton, not yet peer-reviewed but well-designed and preregistered. Worth reading whether or not the “System 3” framing sticks. The behavioral data underneath it is solid.