Tokens: how AIs read text
Why the tiny chunks matter for cost, context, and correctness
Why 'strawberry' breaks AIs
Open the OpenAI Tokenizer in a second tab: [platform.openai.com/tokenizer](https://platform.openai.com/tokenizer). It's free, no sign-in needed.
Copy one of the three prompts below into it, and watch how it splits your text into coloured chunks. Each chunk is a token. The number at the bottom is what the AI actually sees and gets charged for.
Ask ChatGPT how many letter Rs are in the word strawberry. There's a decent chance it'll say two. It won't be trying to trick you. It genuinely doesn't see the word the way you and I do.
AIs don't read letter by letter. They read in bigger chunks called tokens, and once you know how tokens work, a lot of AI's weird behaviour starts to make sense: why long conversations suddenly get expensive, why the same idea in Polish costs three times as much to process, why a chatbot forgets what you said twenty minutes ago, and yes, why it can't count the Rs.
In the next thirty-five minutes we'll take tokens apart, see them live in a real tokeniser, and finish with a habit-set that could easily save you a few pounds a month on any AI project you build.
The quick brown fox jumps over the lazy dog on a summer afternoon.
Szybki brązowy lis przeskakuje leniwego psa w letnie popołudnie.
The zorglemat pinged at 8:47am 🎉 because MDMA-blockchain-quantum said so #vibes.
Compare the three token counts and ask the class:
- Which of these three uses the most tokens? Why do you think that is?
- What happens to unusual or made-up words? Emoji? Numbers?
- If you were charged by the token (which you actually are, on most AI APIs), which of these prompts would be the most expensive per idea communicated?
Ask a pupil to type in their own name, then a made-up nickname. Same length, wildly different token counts. Then try a phrase they use with mates that isn't a standard English word ('slaps', 'lowkey', 'no cap'). Point out that the AI is quietly less efficient at everything young people actually say, because those tokens weren't common in the training data.
What is a token, actually?
The AI doesn't read your prompt as a string of letters. It doesn't even read it as words. Somewhere between the two sits the token.
A token is a chunk of text that the AI has decided is worth treating as one unit. For common English words that ends up being roughly one token per word, plus a token or two for punctuation and spacing. For less common words the tokeniser breaks them into pieces ("tokeniser" itself is three tokens: `token`, `is`, `er`). For non-English text, for emoji, and for made-up words, the token count balloons because each rare piece needs its own separate token.
Every input you send and every output the AI generates is measured, priced, and length-limited in tokens. Not letters, not words. Tokens.
Take a sentence from the last novel or textbook the class was reading. Paste it into the tokeniser at the front. Try three variations: original, ALL CAPS, and translated to another language via any translator. Get the class to predict which will use the most tokens before you press Show.
Three places tokens quietly bite
You might think tokens are a developer's problem. They're not. They quietly shape a lot of everyday AI behaviour.
Before you tap the cards below, have a think: where in your day-to-day AI use might a token count secretly be causing you a problem?
Tap each card to reveal.
Cost per API call
Every commercial AI API charges by the token, with different rates for input and output. A 4,000-token conversation with GPT-4o costs roughly 3-4 pence; do that thousands of times a month and it adds up fast. Prompt hygiene isn't an aesthetic choice, it's a budget line.
The context window
Every model has a maximum token limit for a single conversation (its "context window"). Once you hit it, the AI starts forgetting the oldest parts of the conversation. That's why long ChatGPT threads suddenly seem to lose the plot: they literally have.
Latency + energy
More tokens means more compute, which means slower responses AND more energy per query. A 500-token prompt is faster and greener than a 5,000-token one, even if the answer is the same. This connects tokens to the AI Energy lesson elsewhere on the Lab.
So a well-shaped prompt is cheaper, keeps its memory longer, and hits the planet less hard. Three wins for the price of one habit.
Give each pair one of these scenarios and ask which of the three tokens-matter places is being hit hardest.
- A school chatbot suddenly starts forgetting what a pupil told it two questions ago.
- A start-up's monthly OpenAI bill tripled after they let users paste in whole PDFs.
- A live-demo AI assistant takes ten seconds to reply, embarrassing the presenter.
Good answers will name the dominant driver: context window, cost, latency. Excellent answers will spot that they overlap.
Why AIs can't count letters
Back to the strawberry problem. The reason the AI gets it wrong isn't lack of intelligence, and it definitely isn't malice. It's tokens.
When you type strawberry, the tokeniser doesn't hand the AI ten letters. It hands it something like `straw` + `berry`, which is two chunks. The AI then dutifully counts the Rs it can see in those chunks and reports back. It's not being lazy; it's answering the question it was actually shown, which isn't the question you meant to ask.
Same reason it struggles with anagrams, with reversing words character by character, with counting emoji, with formatting dates that fall on unusual token boundaries. Any task that treats a letter as the unit of meaning is a task tokens quietly sabotage.
Give the class three challenges and ask them to predict which the AI will get right and which it will bungle. Then try live.
- How many letters in the word 'onomatopoeia'?
- Spell 'lasagna' backwards, one letter per line.
- How many emoji are in this sentence: 'I love 🍕 and 🥞 and 🍰 and 🍿'?
All three are letter-level or symbol-level tasks. Pupils should notice a pattern: token-friendly questions get token-friendly answers, and letter-friendly questions do not.
Bad prompt, good prompt
Every token in your prompt costs money, eats context, and slows the response. Every token in the AI's output does the same. So the game is: get the same result in fewer tokens.
The two prompts below ask for exactly the same thing. Both work. One is roughly 4x cheaper than the other. Copy each into the tokeniser to see the difference for yourself.
Hello, I hope you're well today. I was wondering if you would be so kind as to please generate for me, if it's not too much trouble, a fairly short list of maybe about five or so key things that a Year 11 student in the UK might genuinely want to know about the topic of climate change, ideally in a format that is easy to read and reasonably clear. Thank you very much in advance!
~90 tokens, most of them padding. Chatty framing costs the same as substantive content, and the AI often mirrors it back at length. On a paid API this is roughly 3-4x the price of the tight version.
List 5 climate-change facts a UK Y11 student should know. Bullet points, one line each, no preamble.
~25 tokens. Names the audience, the count, the format, and forbids waffle. Same output shape, fraction of the cost. The AI also tends to be MORE useful because the constraints are clearer.
In pairs, hand pupils this bloated prompt and ask them to cut it to under 30 tokens (roughly 20 words). No losing intent. Test the winners in a real AI at the front.
> Please could you give me a fairly detailed and thoughtful summary, of maybe about a paragraph or two in length, of the main causes of the First World War, in a way that is suitable for a Year 10 History homework task, if that would be OK?
The game rewards deleting all the politeness padding, all the hedging, and any redundant instructions.
So what would you do?
Time to put it into practice. You're a Y13 building a small AI chatbot for a business studies project: it'll answer questions about your fictional coffee shop. You've got a free-tier API key with 40,000 tokens per day, and you're expecting about 200 questions.
Do some quick maths: 40,000 tokens ÷ 200 questions = 200 tokens per question total, split between prompt AND response. That's tight. A greeting + a menu list + a polite sign-off could easily blow it in one exchange.
You need a plan.
Split the class into groups of three. Each group gets 200 tokens per question and this brief: build a system prompt + response template that a coffee shop bot could use.
Success = surviving 5 test questions from another group without the AI going over budget or being useless. Pupils will naturally discover: cut the polite framing, use compact wording, abbreviate the menu, put the system prompt on a strict diet.