Notes on the confidence trap
How fluent AI is systematically destroying my ability to think critically and why I won't notice until it's too late.
Back in September 2025, I was in a chilled conference room, the kind with the hermetically sealed windows and the aggressively neutral art, watching at product leadership team about to sign off on a portfolio strategy bet.
The presentation? Honestly, it was a work of art. Beautifully kerning on the slides, a market analysis that felt exhaustive, technical specs that seemed to pre-empt every hesitation I had. It was hypnotic.
I asked exactly one question: “What assumptions would need to be false for this to be a disaster?”
Silence. Just the hum of the ventilation.
It turned out the entire business case was standing on three incredibly optimistic legs. But the Large Language Model (LLM) they used had generated prose so buttery smooth, so undeniably fluent, that questioning it felt like admitting you were the only idiot in the room who didn’t get it. We caught it. This time. But most organizations? They won’t.
And that’s when the cold realization washed over me. The biggest risk with AI right now isn’t that it hallucinates or that it’s biased, or that it breaks a law. It’s that conversational AI is systematically dismantling our ability to think critically and we are paying a monthly subscription for the privilege.
I’ve spent twenty years scaling products to billions of users. I’ve seen tech turn industries inside out. But I have never seen executive teams make such confidently terrible decisions based on such shallow analysis, all while patting themselves on the back for being “data-driven”.
This isn’t about AI being wrong. It’s about AI making us feel certain when we should be terrified.
The cliff we can’t see
I’ve been reflecting over this concept called “death by GPS”, it’s not a metaphor. Years before ChatGPT, researchers were documenting actual drivers who stared at a cliff, heard their GPS say “drive forward,” and drove forward.
They overrode their own eyes. They overrode the environment. The screen said go, so they went.
It’s called automation bias. Goddard, Roudsari and Wyatt in Automation bias: a systematic review of frequency, effect mediators and mitigators found that humans consistently defer to algorithmic recommendations even when those recommendations violently conflict with their own professional judgment.
We are seeing this happen in Strategy, Product, Ops, Finance. All at once. But unlike the car over the cliff, in business, you don’t hit the ground for eighteen months. You’re falling, but you feel like you’re flying.
The seduction of fluency
I need to articulate exactly how this is breaking my brain, because I feel it happening.
Fluent answers feel true. It’s a hack in our cognitive firmware. Think about how we used to search. Google gave us ten blue links. We had to click, read, evaluate, stitch it together. We saw the seams. We saw the disagreement. LLMs hide the seams. You get this clean, authoritative, uninterrupted narrative. No dissenting view. No “confidence interval”. Just text.
Research by Markus Bink showed that how an interface looks shapes our credibility judgments more than the actual source quality; research on Google Search Growth shows that the majority of searches now end without a click. We are consuming conclusions without ever glancing at the evidence.
We spent twenty years optimizing our products for “frictionless” experiences. Well, congratulations to us. We optimized away epistemic caution.
Five ways I’m losing my mind (and my judgment)
I’m watching this happen in real-time.
The Theater of Reasoning: You know that token streaming? The way the cursor blinks and words appear one by one? It mimics human thought. Your brain sees that and thinks, “Wow, it’s working hard”. It’s not. It’s theatre. Those little delays make the system feel thoughtful. I’ve watched senior VPs wait for a three-second pause and say, “See? It really crunched the numbers there”. The pause meant nothing. The perception meant everything.
The Death of Uncertainty: Real experts say “it depends”. They hedge. AI optimizes hedges away because they look weak. Back in 2005, Giles published in Nature, and Chesney in First Monday (2006), showing that people overestimate the completeness of Wikipedia because they never see the “Talk” pages where the editors are fighting. LLMs are Wikipedia without the Talk pages. All reliability, no visible struggle.
The Mirror Trap: The system uses my words back to me. It frames the answer using my assumptions. I feel understood, so I trust the output. It’s just pattern matching, but it feels like alignment. I fall for this constantly.
Narrative Coherence: Human experts stumble. They backtrack. That signals they are thinking. AI prose is a superhighway: smooth, straight, confident. I ran a test with my product teams: same analysis, one presented by a stumbling human, one by a smooth LLM. They rated the LLM as “more thorough” every time. The content was identical.
The Confidence Loop: AI is right often enough that we get lazy. Freibauer’s research in the Journal of Behavioral Finance (2024) on trading apps is the perfect parallel. Simplified interfaces on apps like Robinhood make users gain confidence faster than they gain skill. They trade more, lose more and don’t learn. We are doing this with corporate strategy. We are day-trading our future with unearned confidence.
The Stack Overflow effect
It’s the “copy-paste” culture applied to thinking. Rahman (Mining Software Repositories, 2019) and Jallow (Empirical Software Engineering, 2020) found that insecure code snippets on Stack Overflow get upvoted, copied and spread into thousands of projects. Developers copy what “works” without knowing why.
Now, imagine that but for legal reasoning. For health decisions. For layoffs. We are outsourcing the judgment of “does this solve the right problem?” to a machine that doesn’t know what a problem is.
If doctors can’t resist, can I?
Different systematic reviews show that clinicians defer to algorithms even when they are wrong. These are people with medical degrees. Life and death stakes. And they still cave to the screen, especially under time pressure.
Now, take that deference and apply it to a 24-year-old Product Manager with a deadline. There is no regulator for corporate strategy. We are running a massive, uncontrolled experiment on our collective decision-making ability.
The design decisions we ignore
The “Progressive Token Streaming” and the “Good question!” validation are design choices, not technical requirements, to increase engagement and trust; they are designed to bypass our critical faculties. Compliance teams are out there auditing training data for bias, which is fine, but they should also be auditing the interface for hypnosis.
What do I actually do?
I need to stop spiraling and start acting. Here is my plan. I need to force myself to do this.
1. Audit My Own Epistemics. I need to look at the last five big decisions I made. Did I use AI? Did I accept the framing? I need to find the decisions where my confidence was a 9/10 but the evidence was a 4/10. Those are the landmines.
2. Institutionalize the “Red Team”. Every AI recommendation needs a human whose specific job is to destroy it. “How could this be wrong?” “What evidence contradicts this?” I need to build a process where dissent is mandatory, because the fluency of the AI suppresses natural disagreement.
3. Demand Uncertainty Bars. I am banning the period at the end of a strategic sentence. No more “We should launch in Q4”. It has to be “We should launch in Q4, assuming interest rates hold, with low confidence in our supply chain assumption”. I need to see the math.
4. Competing AIs. Never trust one model. If Claude says X, see what GPT says. If they agree, fine. If they disagree, that gap is where the actual thinking happens.
5. Decision Hygiene. I need a checklist. Not because I’m dumb, but because I’m human. “Have we challenged the prompt’s assumptions?” “Have we looked for disconfirming evidence?”
My expertise is my vulnerability
My experience makes me more vulnerable, not less. My bullshit detector was trained on humans. Humans have “tells” when they are lying or unsure. They fidget. They use weasel words. AI has no tells. It lies with the same calm conviction as it tells the truth. My heuristics are obsolete. Research shows experience mitigates automation bias but it doesn’t cure it. I am not immune.
The choice
I have to choose. I can keep using AI to feel smart, fast and decisive. It feels great. Truly. The dopamine hit of a perfect strategy document generated in seconds is intoxicating. Or, I can use AI to make myself humble. To slow down. To force friction back into the process. The organizations that win won’t be the ones that move the fastest. They will be the ones that realize that “fluency” is not “truth”.
Six months from now, will I be thinking more rigorously or will I just be hallucinating more confidently? I need to answer that. Today.