← All lessons

How AI reads: Tokens and the pieces of language

A library of activities across every year group that makes AI tokenisation visible, testable, and directly connected to the English language curriculum

Filter by key stage or audienceClick one or more chips to filter the sections below.

FAQ for teachers

Common questions teachers ask when running this lesson.

Is tokenisation the same as syllabification or morphology?

No. Tokens are learned from the frequency of chunks in the training data. Morphemes are meaningful parts of words identified by linguistic theory. Syllables are units of pronunciation. All three often coincide, especially for common English words, but they are different things chosen by different processes. The KS2 primary section and the A-level linguistics section tackle this contrast directly.

Why does the same word count as more tokens in one language than another?

Because tokens are learned from frequency in the training data, and English dominated that data. Common English words often become one token each. The same word in Polish, Turkish or Hindi may not appear often enough to earn its own token, so the tokeniser breaks it into several sub-pieces. Users of these languages therefore pay more per query and hit context limits faster.

Do tokens map to letters, syllables, or something else?

None of the above, cleanly. A token is whichever chunk the training process settled on as statistically useful, using an algorithm called byte-pair encoding. A token can be a whole word, a common prefix, a suffix, a couple of letters, or even a single character, depending on how often that chunk appeared in the training data. It is a statistical unit, not a linguistic one.

Why do AI companies charge by token count instead of by word or query?

Because tokens are what the model actually processes. Every token, input or output, costs real GPU time and memory. Words are inconsistent across languages, and queries vary from a single line to an essay, so neither is a fair unit for billing. Tokens are the atomic operation the model performs, so pricing by tokens matches cost to compute more honestly than any word or query rate.

Are tokens the same across ChatGPT, Claude, Gemini and Copilot?

No. Each model family trains its own tokeniser, and the resulting vocabularies differ. A word that is a single token in one model may split into three in another. This is why token counts you see on OpenAI's tokeniser will not exactly match Anthropic's or Google's. For teaching, the concept is universal even though the exact split varies from one product to the next.

Do a model's tokens ever change after it has been trained?

Not without retraining. The token vocabulary is fixed the moment training ends, and any word or symbol that never appeared during training will forever be split into smaller pieces the model does recognise. That is why brand-new slang, freshly-coined product names, or emojis released after the training cut-off often cost surprisingly many tokens. To update the vocabulary, the whole tokeniser has to be rebuilt.

Why does an AI sometimes miscount how many letters are in a word?

Because the model does not see letters, it sees tokens. When asked how many r's are in 'strawberry', a chatbot is really guessing about the internals of tokens like 'straw' and 'berry', which it processes as opaque chunks. Letter-level questions expose the token boundary directly. The workaround is to ask the model to think through the word letter by letter first.

How do I check the tokenisation live without paying for API access?

The OpenAI tokenizer web tool is free and browser-based, so you can point a projector at it and go. Hugging Face also hosts free tokenizer demos for a wide range of models. Neither requires an account, neither charges per use, and both give you the exact token splits you need for the classroom demonstrations in every section on this page.

Common misconceptions

What pupils tend to think, and what to say back.

Pupils often say

The AI reads my sentence word by word, just like I would.

It's actually

It does not. Before doing anything else, the AI chops the sentence into pieces called tokens. Sometimes a token is a whole word, but often it is a common chunk of letters that sits inside many different words. The AI only ever sees the chopped-up version.

Try asking

If the AI is not reading whole words, what do you think the pieces it does read might be?

Pupils often say

A token is basically the same thing as a word.

It's actually

Only sometimes. Short common words like 'the' or 'cat' often are a single token. Longer or rarer words get chopped into several tokens, and something like 'antidisestablishmentarianism' can become half a dozen pieces on its own.

Try asking

Which words in the sentence you just spoke do you think would each be a single token, and which would be more?

Pupils often say

Tokens are just single letters, aren't they?

It's actually

Very rarely. A tokeniser prefers the longest common chunk it can find. Single letters are the last resort, used only when a run of letters does not match anything the tokeniser has already learned as a group.

Try asking

If a tokeniser broke every word into single letters, what would it lose that grouping the letters helps it keep?

Pupils often say

The tokeniser is basically a spell-checker.

It's actually

It is not. A tokeniser has no idea whether your spelling is right. Its only job is to slice the text into the pieces the AI will read next, whether those pieces spell real words or not. It will happily tokenise complete nonsense.

Try asking

If you spelt a word wrong, would the tokeniser tell you, or would it just chop the wrong spelling into pieces and pass it on?

Pupils often say

If I type my name in slowly and carefully, the AI reads it more carefully.

It's actually

Typing speed never reaches the AI. By the time your name gets to it, the text has already been chopped into tokens and stripped of any timing information. The AI reads the pieces, not the pace at which you typed them.

Try asking

What actually reaches the AI once you press send: the timing of your typing, or just the letters you finished with?

Pupils often say

AIs count characters, not tokens.

It's actually

Characters and tokens are two different rulers. Two words that look the same length in letters can be very different in tokens, and it is tokens, not characters, that the AI uses to plan its answer and that services use to charge for it.

Try asking

If two words are both eight letters long but one is four tokens and the other is one, which one costs more to talk to?

Pupils often say

Punctuation is free, only the actual words cost anything.

It's actually

Every full stop, comma, question mark, quote and bracket is a token in its own right. If you type a long string of exclamation marks, a run of dots, or a wall of emojis, you are spending tokens the whole way.

Try asking

If a full stop costs the same as a short word, does that change how you would write a prompt to keep it short?

Pupils often say

Spaces do not cost tokens because they are invisible.

It's actually

In most modern tokenisers the space at the start of a word is part of the token. That is why ' cat' with a leading space and 'cat' without one are two different tokens to the AI, even though they look almost identical to us.

Try asking

If a leading space changes the token, what do you think happens when you paste a sentence with a stray double space in it?

Pupils often say

One token equals one syllable, roughly.

It's actually

They only line up sometimes. 'Cat' is one syllable and often one token. 'Strawberry' is three syllables but usually only two tokens. 'Antidisestablishmentarianism' is twelve syllables and can be around six tokens. Syllables and tokens are made by different rules.

Try asking

What rules make a syllable, and what do you think makes a token?

Pupils often say

Translation is just swapping every English token for the French token that means the same thing.

It's actually

Languages do not share a neat token dictionary. A well-supported language has meaningful chunks the AI recognises; a less-supported language may not, and the AI has to reconstruct meaning from awkward pieces. That is why translation quality varies so much from one language to another.

Try asking

If you tried this on a language the AI has hardly seen, what do you think would go wrong first?

5-minute prep

Five ready-to-run ways to open this lesson. Pick one, copy the prompt, paste it into ChatGPT or Copilot.

Starter5-8 min

Tokenise ten words and count the pieces

The clearest first look at tokens. Ten words go in, a count table comes back, and pupils see straight away that AI does not read the way they do.

I'd like to show a class what tokenisation actually looks like. Please tokenise the following ten words, one after another, and for each word show me (a) the pieces the tokeniser produced with a vertical bar between them, and (b) the total token count. The words are: cat, unhappiness, playground, strawberry, therapist, colour, color, running, antidisestablishmentarianism, Aoife. Present the result as a small table with three columns: word, token pieces, token count. Sort the table from lowest count to highest count. Do not add any extra explanation before or after the table.

Compare8-10 min

Two tokenisers, same ten words

Run the same word list through two different tokenisers side by side. Whatever comes out identically is common ground; whatever differs shows you the ruler each AI is using.

I'd like to compare how two AI tokenisers slice the same words, for a classroom demo. Please tokenise the following ten words twice: once as a modern GPT-style tokeniser would slice them, and once as a simple word-level tokeniser would slice them. The words are: playground, strawberry, unhappiness, therapist, colour, Wojciech, sunbird, running, antidisestablishmentarianism, Ngozi. Present the result as a three-column table: word, GPT-style tokens with vertical bars between the pieces, word-level tokens with vertical bars between the pieces. At the very bottom, give the total token count under each of the two systems. Do not add any commentary before or after the table.

Explain10-12 min

Why does the same word cost different amounts?

Pupils have seen the counts differ. Now ask the AI to explain, in plain classroom English, why the same word turns into different numbers of tokens on different systems.

Imagine you are talking to a curious class of pupils who have just noticed that the word 'strawberry' becomes two tokens on one AI tokeniser and five tokens on another, even though it is spelled the same both times. Please explain, in three short paragraphs of plain classroom English, WHY the same word can produce different token counts. Cover: (1) how a tokeniser learns its chunks from a big pile of training text, (2) why more common words end up shorter and rarer words end up longer, (3) why two AI systems trained on different text piles end up with different rulers. Do not apologise or hedge. Just explain.

Check12-15 min

Tokenise the class register and group by name origin

Turn tokenisation into an evidence exercise. Count tokens per name, group by likely linguistic origin, and let the pattern show you which languages the AI grew up with.

I'm running a lesson on how AI tokenises pupil names. Please tokenise the following list of full names with a modern GPT-style tokeniser and give me a token count for each. Paste the full class register in place of this sentence. Then produce two tables. Table one lists every name with its token count, sorted from lowest count to highest. Table two groups the names by likely linguistic origin (English, other European, South Asian, East Asian, African, Middle Eastern, other) and shows the mean token count per group. Do not add any commentary before or after the two tables.

Repair12-18 min

Design a token-count-fair version of the same task

The advanced move. Once pupils have spotted the unfairness in the counts, ask the AI to redesign the task so the token cost per pupil is as even as possible, then have the class judge whether the fix is real or cosmetic.

In your previous answer you tokenised our class register and names from certain linguistic backgrounds cost several times more tokens than others. I would now like you to redesign the classroom task so that the token cost per pupil is as close to equal as possible, without asking any pupil to change their name or use a nickname. Suggest three concrete changes a teacher could make to how they write a prompt about the whole class (for example, referring to pupils by number, using initials, or asking about groups rather than individuals). For each change, name one thing the pupil loses and one thing the pupil gains.