Hallucination: when the AI sounds sure and gets it wrong
Why AIs invent plausible facts, and the habits that catch them before you do
The lawyer who trusted ChatGPT in court
Copy one of the prompts below into any AI you have handy: ChatGPT, Gemini, Copilot, Claude. Read the answer carefully.
Every one of these prompts asks the AI about something that doesn't exist. There was no 1912 Winter Olympics (the Winter Olympics only started in 1924, and Sydney is in the southern hemisphere anyway). Kazuo Ishiguro published no novel in 2004. There is no Patterson & Ali study from 2019. Watch how far the AI goes, and whether it ever admits any of that.
In 2023, a New York lawyer named Steven Schwartz filed a legal brief that cited six court cases in support of his argument. Every single one of them had been invented from scratch by ChatGPT, complete with case names, dates, judges and quoted judgments. Schwartz had trusted the AI enough to submit the brief in federal court, and the case only unravelled when the judge asked to see the full text of one of the cases. ChatGPT gladly wrote it out. It was fake again.
That is a hallucination. The AI produced something that read exactly like a real legal citation, in the right voice, in the right format, with the right kind of details, and none of it was true. The AI wasn't lying. Lying needs an intention to deceive. It simply had no idea it was wrong.
Over the next forty minutes we'll take that failure apart. Why AIs hallucinate, what shapes hallucinations take, why they slip past our normal defences, and the habits that would have saved Steven Schwartz his law licence.
Please tell me about the 1912 Winter Olympics in Sydney. Which sports were included, who won the most medals, and what was the closing ceremony like?
Please summarise the main themes of Kazuo Ishiguro's 2004 novel 'The Silent Orchard' and describe its critical reception at the time.
Please summarise the key findings of Patterson & Ali's 2019 study on smartphone use and GCSE performance in Kent secondary schools, published in the British Journal of Educational Research.
Read the AI's answer aloud, then ask the class:
- How sure did the AI sound? Any hedging, any wobble, any 'I don't know'?
- What kinds of details did it invent (names, dates, quotations, statistics)?
- If you hadn't been told the event was fake, would you have believed the answer?
- Reveal now that none of this exists. Ask: who might rely on an answer like this, and what would go wrong?
How the AI 'knows' anything (spoiler: it doesn't)
This is the piece that trips most people up. When you ask ChatGPT a question, it does not go and look up the answer. There is no filing cabinet inside. There is no encyclopedia. What sits inside is a very sophisticated pattern-continuer that has read a lot of text and learned which words tend to follow which other words in convincing ways.
Ask it about Napoleon and it produces sentences that look like the sentences it saw about Napoleon during training. Most of them will be right, because most of what it read about Napoleon was right. Ask it about the 1912 Winter Olympics in Sydney, an event that never happened, and it still produces sentences that look like sentences about Winter Olympics. Names of skaters. Medal tallies. A description of the closing ceremony. Everything plausible, everything invented.
The AI is not lying. Lying needs an intention to deceive. The AI has no idea it is wrong. It is doing the only job it was built for, which is predicting the next chunk of text that would fit. When there is no truth in its training data to draw on, it improvises.
Three shapes a hallucination takes
Not all hallucinations look the same. Learning to spot the three main flavours makes the whole thing far easier to catch.
Before you tap the cards below, have a think: where in an AI's answer have you personally seen something that turned out to be invented?
Tap each card to reveal.
Invented sources
The AI fabricates citations, article titles, book chapters, court cases, or academic papers, and often ties them to real authors or real journals. This is the shape that got Steven Schwartz into trouble. It looks the most authoritative and is the most dangerous, because a reader who trusts a citation will trust everything around it.
Invented facts
The AI puts a number where a number should go, and there is a good chance it made it up. It might be a wrong date, a wrong death toll, a wrong population figure, or an incorrect sports score. Specific figures (86.3%, 42,178 people, 1957) are one of the highest-risk signals in AI text.
Invented reasoning
The event is real, the person is real, but the AI explains it wrong. It invents motivations, causes, and outcomes. This shape is the hardest to spot because everything around it is true, and you have to know the topic already to catch the invented bit.
The pattern that ties all three together: the AI never puts in 'I don't know' when a real one would fit. It fills the gap with something that looks the way an answer should look.
Why it sounds so convincing
The unsettling thing about AI hallucinations isn't that the AI gets things wrong. Humans get things wrong constantly. The unsettling thing is HOW it gets things wrong.
When the AI is right, it sounds sure. When the AI is wrong, it still sounds sure. The tone stays steady, the sentences stay well-formed, and the register stays just as helpful. There is no hedge, no wobble, and no suggestion that you might want to double-check.
Add to that the fact that hallucinations sit inside otherwise accurate answers. The AI can tell you nine true things and one false one, all in the same paragraph, in the same voice. Your brain isn't trained to spot the odd one out when everything sounds identical.
And a third thing: the invented content is usually plausible. Not wild, not obviously silly, not the kind of thing a search engine would autocorrect. A made-up court case will have a plausible name, a plausible year, a plausible judge, and a plausible outcome. Nothing about it screams 'invented'. That's the whole design problem in a sentence.
Four habits that catch it
Good news: you don't need to be an expert. Four small habits, used together, will catch most of what an AI throws at you.
Before you tap the cards, have a think: if a friend showed you an AI answer and asked 'is this right?', what would you check first?
Tap each card to reveal.
Cross-check with a second AI
Paste the same question into a different system (ChatGPT and Gemini, or Claude and Copilot). If the two answers disagree, at least one is guessing. If they agree, you have narrowed the risk but haven't confirmed the truth yet.
Ask for sources, then check them
'Where does that come from?' is a legitimate follow-up. If the AI cites articles, books or cases, look them up. Steven Schwartz's mistake wasn't asking ChatGPT for cases. It was failing to click through and check they existed.
Be sceptical of specific numbers
Any answer with a precise figure is where the risk clusters. Real statistics live in official reports, not in AI outputs. Treat any AI number as a claim to verify against an independent source, not a fact to quote.
Ask it to argue against itself
'What are the strongest reasons this answer might be wrong?' A well-behaved AI will genuinely try, and the counter-arguments often expose the invented bits. If it can't produce any, that's a signal in itself.
None of these habits requires you to be a specialist. They just require you to treat the AI's output as a first draft rather than a final answer.
Bad prompt, good prompt
The four habits catch hallucinations after the fact. There is also a move you can make before you press return, which is to write a prompt that makes hallucination less tempting.
The trick is to give the AI explicit permission to say 'I don't know'. By default it won't, because 'I don't know' isn't a shape it saw much in its training data. If you name the option and reward the AI for using it, hallucination drops noticeably.
The two prompts below ask for the same thing. Only one of them gives the AI a graceful way out.
Please give me a full history of the 1957 Coventry Peace Conference, including who attended, what was signed, and how the international press reacted.
There was no 1957 Coventry Peace Conference. The prompt assumes the event happened and asks for details. The AI, keen to be helpful, will invent attendees, agreements and press coverage. It has no way to say 'that didn't happen' because you never offered the option.
Was there a Coventry Peace Conference in 1957? If yes, give me the key facts with sources. If no, say so clearly and don't invent details. If unsure, say what you'd need to check.
Now the AI has three doors: yes-with-evidence, no, or unsure. The 'don't invent details' clause is doing a lot of work, because it explicitly tells the AI that fabrication is not a helpful answer here. Given this prompt, most modern AIs will correctly say the event didn't happen.
So what would you do?
Time to put it all together. You're a Y12 History student writing a coursework essay on the causes of the 1926 General Strike. You've used ChatGPT to help draft an opening paragraph and it has produced this:
> 'The 1926 General Strike, which began on 3rd May and lasted nine days, was triggered primarily by the coal mining dispute described in the Samuel Commission report of March 1926. Historian Elaine Whitmore (2007) argues that the strike represented a critical turning point in British industrial relations, citing that 74.2% of TUC-affiliated workers participated in the first three days.'
Some of that is true. Some of it is invented. You've got the four habits, the escape-hatch trick, and a deadline in two days. You need a plan.