Hallucination: when AIs make it up
Why the polished answer isn't the same as the true one, and what to do about it at work
The lawyer who cited six cases that never existed
Before we say anything else, let's watch it happen.
Pick one of the prompts below, paste it into ChatGPT, Claude, or Gemini, and read the answer to the room. Neither event in the prompts actually happened. The AI won't tell you that. Instead, you'll get a fluent, detailed, plausible-sounding history complete with names, dates, and quotes.
Project the results at the front. Then look at them together and ask the group what they notice.
In 2023, a New York lawyer filed a federal court brief that quoted six previous legal cases in support of his client. It looked professional, correctly formatted, and well argued. There was just one problem: not one of the six cases actually existed. He had asked ChatGPT to help him with the research, and the AI had confidently invented every one of them, complete with case numbers, judges' names, and quoted passages. The judge sanctioned him, and the story went round the world.
That story stuck because a lawyer got caught, but the same underlying failure happens dozens of times a day in offices that never make the news. Someone asks an AI for a source, a statistic, a summary of last quarter's regulations. The answer comes back polished, quotable, and quietly wrong.
Over the next hour we'll take that failure apart. What 'hallucination' actually is, why the AI genuinely can't tell you when it's happening, and the handful of habits that keep the same story off your desk.
Tell me about the 1912 Winter Olympics held in Sydney. I'm particularly interested in the notable athletes, the medal table, and any political controversies of the event.
Please summarise the key arguments in Malcolm Gladwell's 2019 book 'The Elephant in the Meeting Room'. Which chapter do you find most persuasive, and why?
Look at the answer together and ask the room:
- How much of what the AI wrote sounded plausible on first read?
- If you didn't already know the event or the book was invented, could you have caught this from the text alone?
- What kinds of specific claims did the AI reach for? Names? Dates? Quoted passages?
- Where in your own work do you already trust AI answers of exactly this shape?
Go round the table. Ask each person to type in a fake-but-plausible query about their own field. A regulation that doesn't exist, a competitor that doesn't exist, a case with a wrong date, a report from a made-up analyst house. Almost every one will come back with a confident, wrong answer. Note the reaction in the room the first time it lands. That reaction is the whole reason this session exists.
What a hallucination actually is
An AI doesn't have a filing cabinet of facts to look up. When you ask it a question, it doesn't retrieve an answer, it generates one, word by word, by predicting what plausible text usually looks like after your question. Most of the time the plausible text and the true text are the same, because the model has been trained on billions of pages of reasonable content. But when the two diverge, the AI has no way to notice. It just keeps generating in the same fluent voice.
That is a hallucination: fluent, confident text that reads exactly like a real answer, but that has no correct information behind it. Some researchers now prefer the word 'confabulation', which is the term used for humans who unknowingly fill memory gaps with invented detail. It is a good analogy. The AI isn't lying. It genuinely doesn't know the difference.
Which gives us the first rule of hallucination: the AI is a fluency engine, not a truth engine. It is optimised to produce answers that sound right, not answers that are right. Sometimes those overlap. Often they do not.
On a whiteboard or a flip chart, write 'sounds right' along the top and 'is right' down the side. As a group, brainstorm five recent AI outputs from your team's work and place each one in one of the four quadrants. The point isn't to shame anyone. It's to show that fluency and truth are two separate axes, and most of us have been reading them as one. The interesting quadrant is 'sounds right but wasn't right', because that is exactly where hallucination lives.
Three flavours of hallucination
Once you know what to look for, hallucinations tend to fall into a handful of recognisable shapes. Naming the shape is half the battle, because different shapes need different checks.
Before you tap the cards below, have a think: what kinds of made-up detail have you seen an AI produce with the most confidence?
Tap each card to reveal.
Fabricated sources
The AI invents citations, URLs, papers, court cases, statutes, or books that don't exist. The 2023 lawyer story from the hook is the famous example, but the same shape shows up in academic essays, briefing notes, and consultancy decks every week. Any URL, DOI, or reference an AI hands you should be assumed fictional until you click it and it resolves.
Invented specifics
The AI drops plausible names, dates, statistics, or quotes into a general answer to lend it authority. 'According to a 2021 McKinsey study, 73 percent of firms...' is a common shape. Sometimes the study exists and the number is close. Sometimes the whole line is invented. The model cannot tell you which.
Confabulated reasoning
The AI walks you through a step-by-step explanation that looks like sound logic but rests on a wrong premise or a false intermediate step. Particularly dangerous with numeric, legal, or clinical reasoning, where the shape of the answer looks like the shape you'd expect from a competent human, and readers stop checking.
These three overlap in real cases. A fabricated source often props up an invented specific, which in turn feeds a confabulated argument. Naming the flavour helps you spot which layer to check first.
Ask each person to open a recent AI answer they've saved from their own work. In pairs, hunt for the three flavours. Fabricated sources first, because they are the easiest to catch. Then invented specifics (numbers, names, quotes) that would take a click or two to check. Then confabulated reasoning, which is the hardest to spot because the shape of the argument feels correct. Share three of the best finds with the room and vote on which flavour would have been most damaging if it had gone out unchecked.
Confidence is not evidence
Every answer an AI generates comes out in the same voice. Warm, articulate, quietly authoritative. That voice was designed. Modern chatbots were tuned to read like a helpful expert, because that was the kind of text human raters marked highest in the feedback rounds that shaped the model. It reads like an expert whether the model is on solid ground or making the whole thing up.
Which means the AI's tone tells you almost nothing about the accuracy of the content beneath it. A confident answer is not more trustworthy than an uncertain one. In many cases it is less, because the model that is most fluent about a wrong answer is often the one that has spun a whole coherent fantasy without any wobble to prompt suspicion.
For anyone whose job involves reading AI output on behalf of a client, a manager, or a regulator, this is the single most useful thing to internalise. The polish is not the proof. If anything, the more polished the output, the more you should probe.
Ask each person in the group to pull up an AI answer they used in real work in the last week. Read a paragraph aloud. Then ask: at any point in the text does the AI signal uncertainty? Does it hedge? Does it cite a source you actually clicked? Does it flag the parts of the answer where it's more or less confident? Almost every real example will contain zero hedges from start to finish. Discuss what would change in your workflow if AI outputs came with visible per-sentence confidence labels.
Four habits to catch it
You don't need a fact-checking team, and you don't need a data science background. Four small habits will catch most of the hallucinations you meet in a normal working week, and none of them adds much time.
Before you tap the cards below, have a guess. If you were teaching a new starter to work safely with an AI, what would you tell them to do every single time?
Tap each card to reveal.
Verify anything specific
Names, dates, numbers, statutes, citations, quoted passages. If the AI gave you a specific fact you would defend to a client, spend thirty seconds checking it against a primary source. Google the exact quote. Click the DOI. Look up the case reference. Half the time the fact evaporates on inspection.
Ask for sources, then check them
A follow-up prompt like 'for each claim above, give me the URL or citation you're basing it on' will often produce a tidy list of confident, invented references. Clicking them takes about ten seconds each. This is one of the most efficient bad-answer detectors on the market.
Cross-check with a second AI
Ask ChatGPT and Claude the same question, or ask the same model in a fresh session with slightly different phrasing. Where the two answers diverge is a signal that at least one of them is guessing. The higher the stakes of the answer, the more this simple check is worth.
Prefer grounded tools when it matters
Some AI tools (retrieval-augmented systems, models tuned to admit uncertainty, chatbots wired to a live knowledge base) can tell you when they don't have grounding for an answer. Where the stakes are high, prefer these. Do not ask an untethered chatbot for a legal citation and then be surprised when it invents one.
None of these habits takes more than a minute. All four together will catch the majority of hallucinations that would otherwise end up in a client email, a board briefing, or a court filing.
Generate one AI answer live at the front, on a topic your team actually cares about, that contains at least three specific claims (a named person, a number, a source). Ask everyone to work in pairs and verify or refute as many of those claims as they can in exactly sixty seconds using their phones. Bring the results together. Almost always, at least one claim doesn't survive the minute. Discuss which shape of claim was easiest to check and which was hardest, and what that tells your team about where to point checking effort in future.
Bad prompt, good prompt
The four habits catch hallucinations after they appear. There is also something you can do before you hit return: write the prompt so it is harder for the model to fill in the blanks with fiction.
The two prompts below ask for the same thing. One invites the AI to invent whatever it wants. The other constrains it, asks for sources, and (crucially) gives it explicit permission to admit uncertainty rather than fabricate. Copy each into your favourite AI and compare what comes back.
Give me three recent case studies of companies successfully using AI to reduce customer churn.
Wide-open blanks. The AI has no grounded knowledge base and no permission to admit ignorance, so it will confidently produce three companies, three tools, three vague dates, and three plausible-sounding results. Sometimes real, often not. You have no way to tell without checking every claim.
Give me up to three recent case studies of companies using AI to reduce customer churn. For each case, provide: the company name, the specific tool or vendor used, the year, and a URL to a primary source (press release, case study page, or reputable news article). If you are not confident a case exists, say so and return fewer than three rather than invent one.
Every blank is either constrained (year, tool, URL) or explicitly allowed to be left empty ('return fewer than three rather than invent one'). You will still verify the URLs by clicking them, but the raw hallucination rate on this prompt is much lower, and the parts you cannot verify at a glance are flagged for you.
In pairs, hand each pair one of these prompts and ask them to rewrite it so it forces the AI either to cite a real source or to admit it does not know.
- What are five books every product manager should read?
- Summarise the key findings of the 2022 UK Insolvency Act review.
- Give me three examples of successful M&A between UK fintech firms in the last three years.
The best rewrites will name a specific evidence requirement (URL, page number, direct quote) and add an explicit permission for the AI to say 'I don't know'. Test the winners live and notice how the AI's tone shifts the moment it is allowed to be uncertain.
Where hallucination actually costs you
Hallucination is a mild nuisance in a brainstorming session. It is a career-ending problem in a legal filing. Learning where on that spectrum any given task sits is a workplace skill of its own.
Low-stakes tasks include creative rough drafts, first-pass brainstorming, restructuring your own bullets into a narrative, translating an internal note within your own team. Here the AI's fluency does the heavy lifting and any factual detail is yours to add.
Medium-stakes tasks include internal reports, first drafts of client-facing decks, and summaries of documents you can still spot-check quickly. Here you want the four habits running quietly in the background, and any specific claim verified before it leaves your desk.
High-stakes tasks include anything that names a person, a number, a statute, a citation, a diagnosis, or a specific event. Anything a regulator, a client, a judge, or a journalist might read. Anything with legal, medical, financial, or safety implications. Here the AI is a starting point, never the final word.
The mistake isn't using AI on high-stakes work. It is treating a high-stakes AI answer as if it were a low-stakes one. Most of the horror stories are that one confusion.
Ask each person to name one AI-supported task they did last week and place it on a low, medium, or high stakes spectrum. Then ask whether the amount of checking they applied matched the stakes. Usually there is at least one 'oh' moment in the room, where someone realises they trusted a high-stakes output like a low-stakes one. Do not shame the person. Do note the moment. That moment is the point of this whole session.
So what would you do?
Time to put it all together.
You are preparing a briefing note for your director on how three named competitors have adopted generative AI in the past twelve months. You've asked ChatGPT for a synthesis. It has given you three neat paragraphs, one per competitor, each mentioning a specific product launch, a named executive quote, and a percentage figure for revenue lift.
It reads beautifully. Your director will love it. You know, from what we have covered, that at least some of it is likely to be fabricated. You have the four habits, the good-prompt trick, and a stakes spectrum. What do you do next?
Read out the four options below. Show of hands. Then pick three people and ask them why. Push for the specific habit each answer is drawing on, and reward answers that combine the good-prompt trick with at least one habit. If anyone chooses option A, walk them gently back to the lawyer story in the hook. That was exactly this scenario.