Tokens: how AIs read your words
Why AIs can't count the Rs in 'strawberry', and what that tells us
Ask ChatGPT how many Rs are in 'strawberry'
Before we say anything else, let's watch tokens happen live.
Open a second tab and go to platform.openai.com/tokenizer. It's free, no sign-in needed. Whatever you paste in, it splits into coloured chunks and tells you how many tokens the AI would see.
Try one of the prompts below. Project it at the front so the whole class can see the colours change.
Try it. Type: How many letter Rs are in the word strawberry? into any chatbot you like.
There's a really good chance it'll say two. It isn't trying to trick you and it isn't being lazy. The AI honestly cannot see the word the way you and I do.
Over the next half hour we'll work out why. It turns out the AI doesn't read your writing letter by letter, or even word by word. It reads in chunks called tokens, and once you know how tokens work, a lot of the AI's weird behaviour suddenly makes sense: why it forgets what you said earlier in a long chat, why the same idea in Polish or Urdu 'costs' more, and yes, why it can't count the letters in a fruit.
The football match on Saturday was really good, especially the last ten minutes.
Sobotni mecz piłkarski był naprawdę dobry, zwłaszcza ostatnie dziesięć minut.
That skibidi lesson was actually kinda peak 😂 #vibes
Look at the three token counts together and ask the class:
- Which one uses the most tokens? Which uses the fewest? Did that match your guess?
- What happens to emoji, hashtags and slang? Are they cheap or expensive?
- If the AI charges by the token (which it usually does), which one of these would cost the most to send?
Ask a pupil to type in their own first name, then a nickname their mates use. Same length in letters, often wildly different token counts. Try a phrase from your form group's Snap streaks or WhatsApp ('no cap', 'lowkey', 'delulu', whatever's current). Point out that the AI is quietly clunkier with anything that wasn't common in its training data, which includes a lot of what teenagers actually say.
So what actually is a token?
When you send a message to an AI, it doesn't line up your text letter by letter. It also doesn't read it word by word. Somewhere in between sits the token.
A token is a chunk of text that the AI has decided is worth treating as a single piece. For common English words that's usually one whole word per token, plus a token for the space or punctuation. For rarer words the tokeniser chops them into smaller bits. The word strawberry, for example, gets split into something like `straw` + `berry`, two chunks instead of one.
Every message you send AND every reply the AI gives back is counted in tokens. Not letters. Not words. Tokens are the unit the AI actually pays attention to.
Take one sentence from whatever book the class is reading in English. Paste it into the tokeniser at the front. Then paste three versions of it: the original, the same sentence in ALL CAPS, and the same sentence run through Google Translate into a language nobody in the room speaks. Ask pupils to predict which will use the most tokens before you show the count. The all-caps and translated versions almost always balloon.
Three ways tokens quietly matter
Tokens sound like something only coders would worry about. They're not. They quietly shape a lot of what you see when you use any AI.
Before you tap the cards below, have a guess. Where do you think a token count might secretly be causing a problem in your everyday AI use?
Tap each card to reveal.
Cost
Every commercial AI charges by the token, with different prices for what you send in and what it sends back. That's usually invisible to you on free ChatGPT, but it's why 'AI credits' run out on paid apps and why long chats eat them faster than short ones.
Memory (the context window)
Every AI has a maximum number of tokens it can hold in a single conversation. Once you hit that limit, the oldest parts of the chat drop off the back. That's why a really long ChatGPT thread suddenly starts forgetting things you told it near the start.
Speed
More tokens means more work for the AI, which means a slower reply. A tight 200-token question is much faster to answer than a bloated 2,000-token one. If your AI feels sluggish, the length of the conversation is often the reason.
So keeping your messages tight isn't just polite. It saves money, it stops the AI forgetting things, and it makes replies faster. Three wins from one habit.
Read out these three real situations and ask the class which of the three tokens-matter places is being hit hardest.
- A pupil is chatting to ChatGPT about a History essay. After about an hour it starts giving answers that ignore what they said at the start.
- A group is building a free-tier AI chatbot for a school project and they've been told they only get 40,000 tokens a day.
- An AI drawing app takes ages to reply when the pupil pastes in three paragraphs of description.
Good answers name the main driver: memory, cost, speed. Great answers spot that two of these often happen together.
Why AIs can't count letters
Back to the strawberry problem. The reason the AI gets it wrong isn't that it's stupid, and it definitely isn't being cheeky. It's tokens.
When you type strawberry, the tokeniser doesn't hand the AI ten letters. It hands it two chunks, roughly `straw` and `berry`. The AI then counts the Rs in what it can actually see, which is those chunks. It answers the question it was really shown, which isn't quite the question you meant to ask.
Same reason it fumbles anagrams, gets confused if you ask it to spell a word backwards one letter at a time, or miscounts emoji in a sentence. Anything where the letter (or the individual symbol) is the important thing is a task tokens quietly wreck.
Give the class three challenges. Ask them to predict which the AI will get right and which it will bungle. Then try them live at the front.
- How many letters are in the word 'onomatopoeia'?
- Spell 'raspberry' backwards, one letter per line.
- How many emoji are in this message: 'I love 🍕 and 🍰 and 🍿'?
All three are letter-level or symbol-level tasks. Pupils should notice a pattern: token-friendly questions get token-friendly answers, and letter-friendly questions do not.
Which words are cheap, which are pricey?
Not all words cost the same in tokens. The tokeniser was trained on a mountain of text (mostly English, mostly from the internet), so it has neat little chunks ready for anything common. Anything unusual has to be built up from smaller pieces.
Have a quick guess before you look at the match below: which do you think uses more tokens, the word the or the word antidisestablishmentarianism? And what happens if you type your friend's name into the tokeniser?
In pairs, get pupils to guess (before checking) which of these will use MORE tokens: a) 'football' vs 'obfuscation', b) their own first name vs a random string of letters like 'kzjqxlp', c) 'hello' vs 'skibidi'. Then check each pair in the tokeniser. Draw out the rule: common = cheap, rare = pricey. The AI is quietly biased towards things it has seen a lot before.
Bad prompt, good prompt
Every extra token in your message costs money, chews up memory and slows the reply. Every extra token in the AI's reply does the same. So the game is simple: get the same result in fewer tokens.
The two prompts below both ask for exactly the same thing. Both work. But one is roughly three times cheaper to send, and often gives a better answer because the instructions are clearer. Paste each one into the tokeniser and see the count for yourself.
Hi there, I hope you're doing OK today. I was just wondering, if it's not too much trouble, whether you might be able to please give me maybe five or so quick tips or ideas for how a Year 8 student in the UK might revise for their end of year Geography test, ideally in a way that's fun and not too boring. Thanks so much!
Roughly 80 tokens, most of which is padding. All the 'please' and 'if it's not too much trouble' and 'thanks so much' costs the same as the useful part. The AI often mirrors your chatty style back at length too, which makes the reply expensive as well.
5 revision tips for a UK Y8 Geography end-of-year test. Bullet points, one line each, no waffle.
Roughly 25 tokens. It names the audience (Y8, UK), the subject, the count, the format, and it tells the AI not to pad the answer. Same job, a third of the cost, and usually a sharper reply.
In pairs, give pupils this bloated prompt and ask them to cut it down to under 25 tokens (roughly 15 words) without losing what it's asking for. Test the shortest winners live.
> Please, if it's alright, could you kindly write a fairly detailed and thoughtful summary, of maybe a paragraph or two in length, of the main causes of World War One, in a way that would be suitable for a Year 9 History homework, if that would be OK for you?
The game rewards deleting every polite word, every hedge, and every instruction the AI would have followed anyway.
So what would you do?
Time to put everything together. Imagine you're building a small AI helper for a school project. It's supposed to answer quick questions about the plot of Romeo and Juliet for younger pupils.
You've been given a free-tier account with 30,000 tokens a day and you're expecting about 100 questions. Do some quick maths: that's 300 tokens per question, split between what you send AND what the AI sends back.
On your first test, you get: 'How many Ss are in the name Rosencrantz?' from a pupil. The AI says 3. You count. It's 2. And your bloated system prompt has already used up half your budget for that message.
You need a plan.
Read the four options below out loud. Hands up on which one they'd pick. Then ask three pupils to explain: one for the strongest option, one for a tempting but weak option, and one for whatever the room actually voted for. Push them to say which bit of the lesson they're using: the token budget, the strawberry problem, or the bad-vs-good prompt rule.