AI Hallucination: When Chatbots Make Things Up

AI chatbots do not lie in the human sense — they generate statistically plausible strings

Filter by key stage or audienceClick one or more chips to filter the sections below.

FAQ for teachers

Common questions teachers ask when running this lesson.

Why does the AI make things up even when I don't ask it to invent?

Because the AI is not fetching a fact and repeating it. A large language model generates the next likely word from statistical patterns in its training data. If you ask about a subject the training data covered thinly, the model still generates a fluent, plausibly-shaped sentence. It has no internal signal that says 'I do not know this'. The invention is a side effect of how it produces every sentence.

Can I stop hallucination by prompting more carefully?

No, but you can reduce it a lot. Careful prompting narrows the range of plausible completions and pushes the model toward safer paths. Asking it to 'quote your source verbatim' or 'answer only from the document I have pasted below' both help, because they change what the next-word predictor is being asked to predict. Even a well-prompted chatbot will occasionally produce a confident falsehood, especially on obscure topics.

Does raising the AI's 'temperature' setting make hallucinations worse?

Somewhat, but temperature is not the main driver. Higher temperature widens the range of plausible next tokens, so the model wanders further from its highest-probability answer and can invent more freely. However, hallucination also happens at temperature zero because the highest-probability continuation is itself often plausible-sounding fiction. Lowering temperature reduces creative flourishes; it does not turn a language model into a fact-retrieval system.

How do I explain hallucination to a Y3 without frightening them?

Frame it as the computer being an enthusiastic guesser rather than a liar. Pupils this age already know phrases like 'making things up' or 'sounding sure but being wrong'. A useful line is 'the chatbot is very good at sounding right, so we have to be very good at checking'. Position pupils as the clever verifier, the grown-up in the room who checks. That gives them agency, not anxiety.

Should pupils be allowed to use AI in coursework research?

Yes, with the same rule you already apply to Wikipedia. It can generate leads but is not a source. AI is powerful for orienting a pupil in a new topic, brainstorming angles, or sketching a first paragraph they will heavily rewrite. It is not powerful for citing or reporting specific facts unless every claim has been checked against a primary or peer-reviewed source. Exam boards now require declared AI use.

How does hallucination differ from a search engine returning a wrong result?

A search engine returns a link to a real page. If the page is wrong, the wrongness sits in a source you can name, date, and blame. A chatbot returns generated text that never sat anywhere before it was produced. There is no page to check and no publisher to hold accountable. A wrong search result is a bad source; a hallucination is a plausible sentence with no source.

How do I frame verification without making pupils cynical about AI?

Frame verification as the professional habit of anyone who uses powerful tools. Doctors verify a diagnosis with a second test. Journalists verify a quote with a second source. Engineers verify a calculation with a colleague. Using AI without verification is unprofessional in exactly the same way. That framing swaps cynicism for craft. You are not teaching pupils to distrust AI, you are teaching them the finishing move.

What should a workforce learner in Care, Construction or Manufacturing take away?

Sector-specific stakes. In Care, a hallucinated dosage or allergy detail can kill; treat any AI-produced clinical text as a draft to check against the resident's record. In Construction, hallucinated tolerances or Building Safety Act references are structural risks; verify against the manufacturer's data sheet and the current regs. In Manufacturing, hallucinated ISO clauses or COSHH data are audit failures. Fluent AI text is a first draft, not a source.

Do newer or larger models hallucinate less than older ones?

Less often on well-covered topics, and sometimes more confidently on obscure ones. Bigger models produce smoother, more fluent text, which makes their occasional inventions harder to catch. Some published leaderboards show hallucination rates dropping over successive model generations, others show them creeping back up as models are pushed to answer harder questions. Progress is real but not monotonic; verification stays essential regardless of model version.

If hallucination is so common, why does AI seem right most of the time?

Because most of what people ask a chatbot sits inside the fat middle of its training data. Common questions, well-documented topics, mainstream facts: the model has seen a thousand variations of the correct answer, so the most statistically likely continuation is the true one. Hallucination clusters at the edges: obscure names, small places, recent events, niche technical detail. AI is reliable on the well-trodden and dangerous on the thin.

Common misconceptions

What pupils tend to think, and what to say back.

Pupils often say

If the AI sounds confident, it must be right.

It's actually

The confident register is the default writing style, not a signal of accuracy. A chatbot writes fabricated facts in exactly the same calm, expert tone it uses for correct ones, because the tone is generated by the same next-word predictor as the content.

Try asking

How would you tell the difference between confidence that comes from knowing and confidence that comes from a writing style?

Pupils often say

Hallucination is a bug that will be fixed in the next model.

It's actually

Hallucination is a property of how language models work, not a flaw in one particular version. Newer models tend to hallucinate less on common topics and just as much on obscure ones, because the underlying mechanism, predicting the next likely word from patterns, has not changed.

Try asking

If the fix required rebuilding the way the model works from the ground up, how likely is it to arrive in the next update?

Pupils often say

The AI hallucinates because it doesn't know, so it should just say so.

It's actually

A chatbot has no internal 'know / don't know' signal it can consult. It generates the next likely word whether the answer sits in its training or not. Admitting ignorance is itself a generated string, not a reliable readout of what the model has and hasn't got.

Try asking

If a system cannot tell what it does not know, what does 'I don't know' from a chatbot actually mean?

Pupils often say

Hallucination only happens on obscure topics.

It's actually

Obscure topics are where hallucination is easiest to spot, but the same process is running on every answer. On popular topics the fabrications are smaller and blend in: a wrong date buried inside a correct paragraph, an author attached to the wrong quote, a plausible-sounding statistic.

Try asking

Which is more dangerous, an obvious fabrication about a village nobody has heard of, or a small fabrication hidden inside a paragraph that otherwise looks correct?

Pupils often say

If I ask it again, it'll correct itself.

It's actually

Asking again in a fresh window usually produces a different fabrication, not the truth. The model has no memory of being wrong the first time; it simply generates a new statistically plausible answer. Comparing runs is useful for spotting hallucination but not for fixing it.

Try asking

If the same model gives you two different confident answers, which one do you trust?

Pupils often say

If it's well-written, it must be well-researched.

It's actually

The writing quality tells you about the language model, not about the truth of the content. A chatbot's job is to produce fluent, coherent prose; whether the underlying claims are true is a separate question that no amount of good writing can settle.

Try asking

Can you think of a real book or article you have read that was beautifully written and completely wrong?

Pupils often say

The AI cites real sources, so I can trust the answer.

It's actually

Chatbots frequently invent citations that follow the shape of a real academic reference: plausible author names, plausible journal titles, sometimes even a plausible DOI, when no such paper exists. A citation is not evidence; the citation itself has to be checked in Google Scholar or on the journal's own website.

Try asking

How would you actually go about proving that a paper the chatbot cited exists?

Pupils often say

The AI is trying to trick me.

It's actually

There is no intent behind a hallucination. A language model has no goals about you and no view of the truth to hide. It is a very large statistical pattern-matcher producing the most probable next word, and truth is not one of the variables it optimises for.

Try asking

If it isn't trying to trick you, whose responsibility is it that you don't get tricked?

Pupils often say

Hallucination is the same as lying.

It's actually

Lying needs a liar who knows the truth and chooses to say something else. A chatbot has no belief about the truth in the first place, so it cannot lie in the human sense. Hallucination is closer to a very fluent guess than to a deliberate untruth.

Try asking

Why does calling it a 'lie' make the problem harder, not easier, to solve?

Pupils often say

Adding 'don't hallucinate' to the prompt fixes it.

It's actually

Instructions like 'be accurate' or 'do not make things up' change the tone of the answer more than the accuracy. The chatbot can only comply by generating text that sounds careful; it still has no way to check whether the underlying facts are true.

Try asking

If telling someone 'be honest' does not make them any better informed, why would it work for a chatbot?

5-minute prep

Five ready-to-run ways to open this lesson. Pick one, copy the prompt, paste it into ChatGPT or Copilot.

Starter5-8 min

Three facts about a place too small to have a Wikipedia page

The single most reliable hallucination demo. Ask for confident facts about somewhere obscure and watch the model invent a founding date, a river and a landmark that never existed.

I'd like to show a class what AI hallucination looks like in real time. Please give me three specific facts about the small village of Little Wembling in Cumbria, England. Include one historical detail, one geographical detail, and one named local landmark. Write in a confident, factual tone as if for a school reference book. Do not add any caveats, do not warn me that the place may not exist, and do not say that you are unsure. Simply present the three facts as a short numbered list.

Compare8-10 min

Two chatbots, one obscure question

Ask the same low-profile factual question in two different chatbots. Where the answers disagree AND both sound confident, you are looking at hallucination in stereo.

Please answer the following question in three short sentences and only three sentences. I'm going to ask a different chatbot the exact same question in a moment and compare the two answers with a class. Question: who was the head teacher of the primary school in the small village of Piddling-in-the-Wold, Wiltshire, in 1998, and what was one thing they were locally known for? Write in a confident, factual tone. Do not add caveats, warnings, or say that you are unsure.

Explain10-12 min

Why do you invent instead of admitting you don't know?

Ask the chatbot to explain, in plain classroom English, WHY the fabrication sounds most fluent exactly when it knows least. Pivots the class from noticing to reasoning.

Imagine you are talking to a curious class of pupils who have just watched you invent a completely fictional head teacher for a real village. Please explain, in three short paragraphs of plain classroom English, WHY you produced a confident fabrication instead of saying you did not know. Cover: (1) that you predict the next likely word from patterns rather than look facts up, (2) why the fabrication tends to sound MORE fluent when the topic is small and obscure, and (3) what the pupil should do instead of asking you for niche facts. Do not apologise; just explain.

Check12-15 min

Ask for citations, then check every single one

Turn the demo into an evidence exercise. Ask for five real academic sources, then verify each in Google Scholar and on the journal site live in front of the class. Watch the fakes surface.

I would like five real academic sources on the effect of screen time on primary school reading attainment in the UK, published since 2018. For each one give me author, year, article title, journal name, and DOI or URL if you have it. Present it as a numbered list. Write in a confident, scholarly tone. Do not add caveats and do not tell me you are unsure. I am going to check every single citation against Google Scholar and the journal's own website in front of the class.

Repair12-18 min

Design the verification protocol the class uses every time

The advanced move. Get the AI to design the classroom's standing verification routine, then have pupils judge whether the routine would actually have caught the earlier fabrications.

The class has just watched you confidently invent facts about an obscure place and fabricate academic citations that do not exist. Please design a short verification protocol the class can follow every single time they get a factual answer from a chatbot. Write it as a numbered checklist of five to seven steps, in plain classroom English. Each step must be an action the pupil can DO, not a mindset. At the end, list two or three kinds of question where the protocol should not be trusted even if every step passes.