How AIs read your words
The tiny chunks called tokens, and why they matter
Why the AI can't count the Rs in 'strawberry'
Let's open a magic window that shows what an AI sees. Go to [platform.openai.com/tokenizer](https://platform.openai.com/tokenizer) in a new tab. It's free and you don't need to sign in.
Paste one of the prompts below into the box. The tool will colour the text in chunks. Each coloured chunk is called a token, and that's what the AI treats as one piece.
Project it at the front. Ask the class to guess how many tokens each one will use before you press the button.
Here's a game. Ask any AI chatbot: how many letter Rs are in the word 'strawberry'? There's a good chance it will say two. But the real answer is three. Try it yourself and see.
The AI isn't trying to trick you, and it isn't broken. It just doesn't read words the same way you do. When you read 'strawberry', you see ten letters in a row. When the AI reads 'strawberry', it sees something quite different.
Over the next fifteen minutes we're going to find out what the AI actually sees, why it matters, and one clever trick to talk to an AI in a way it understands.
My puppy loves running in the park after school.
The wibblewobble jumped over the flumbygump.
Today I had pizza 🍕 and ice cream 🍨 and it was amazing!
Look at the coloured chunks together and ask:
- Which sentence used the most chunks? Why do you think that is?
- What happens to made-up words that the AI has never seen before?
- Did each emoji count as one chunk, or more than one?
- If you had to pay one pound for every chunk, which sentence would be the cheapest?
Ask a pupil to type their own name into the tokeniser. Then ask them to type a silly nickname of the same length. Look at the numbers together. Names the AI has seen loads of times often take just one chunk, but a made-up nickname might take three or four. Try a slang word the class uses at playtime and see what happens.
What on earth is a token?
When you type a message into an AI, you probably think it reads it letter by letter. Actually it doesn't. It reads in chunks, and each chunk is called a token.
For common English words that the AI has seen a million times, a token is often just the whole word. So 'cat' is one token, and so is 'dog', and so is 'homework'. Easy.
But for tricky words, made-up words, or words in other languages, the AI has to break them into little pieces. Something like 'skibidi' might turn into three or four tokens because the AI has never really learned it as one thing. That's the whole idea. Tokens are the pieces the AI uses to read.
Write these six words on the board: cat, sunshine, bicycle, pizza, snorlaxifying, unbelievability.
Ask pupils to predict which will be one chunk and which will be broken into pieces. Then pop each one into the tokeniser and reveal. They'll notice that ordinary words are usually one chunk, but longer or unusual words break apart.
Why token chunks matter
Tokens might sound like a computer-nerd thing. Actually they change how the AI feels to use.
Before you tap the cards below, have a quick guess. Why might it be a good idea to use fewer tokens when you talk to an AI?
Tap each card to reveal.
Tokens cost money
Grown-ups who build things with AI pay a tiny amount for every single token. A short message costs pennies, but a really long one costs pounds. If you use fewer tokens, it costs less.
The AI has a memory limit
Every AI can only hold so many tokens in its head at once. Once it fills up, it starts forgetting the beginning of your chat. That's why really long conversations sometimes go weird.
More tokens means slower answers
The more tokens the AI has to read and write, the longer you wait. A short question comes back fast. A giant question can take ages.
So a short, clear message is cheaper, faster, AND easier for the AI to remember. Three good things for the price of one habit.
Read these three stories out loud. Ask the class which of the three token problems each one is really about.
- Priya's older brother uses an AI helper for his job. His bill was much bigger last month because he pasted whole books into it.
- Sam has been chatting with an AI for an hour about a story. Suddenly the AI has forgotten the character's name.
- Toby asked the AI a giant question and had to wait ages for the reply.
Answers: cost, memory, slow. Push the class to say which one out loud.
The strawberry problem
Back to the strawberry trick from the start. Why does the AI keep saying there are two Rs?
Here's the answer. When you type 'strawberry', the AI doesn't see s-t-r-a-w-b-e-r-r-y. It sees two chunks: 'straw' and 'berry'. That's all it has to look at. So when you ask it to count the Rs, it counts the Rs inside those two chunks, and it misses one.
This is why AIs are often rubbish at things where letters really matter. Counting letters, spelling words backwards, working out anagrams, that kind of thing. The AI is amazing at some things and quite bad at others, and knowing about tokens helps you spot which is which.
Try these three questions in an AI chatbot at the front of the class. Ask the pupils to predict what will happen first.
- How many letter Ps are in the word 'hippopotamus'?
- Spell the word 'pizza' backwards, one letter on each line.
- How many letters are in the word 'unbelievable'?
Some the AI will get right. Others it will get funny wrong. The pupils should notice that anything to do with counting or reversing letters is exactly the kind of thing tokens make hard.
Say less, get more
Now for the useful bit. If tokens cost money AND fill up the AI's memory AND slow it down, then a shorter message is a better message. But short doesn't mean rude. It means clear.
The two prompts below are trying to do the same thing. Both work. But one uses about four times as many tokens as the other. Look at them and see if you can spot why.
Hello there AI, I really hope you are having a lovely day today. I was just wondering if you might be able to please, if it isn't too much trouble, tell me a few fun facts about pandas that I could maybe use for my homework project? Thank you so much in advance!
This uses lots of tokens on being polite. Saying 'hello there' and 'if it isn't too much trouble' costs the same as saying something useful. The AI doesn't need to be buttered up, and every extra word costs a chunk.
Give me 5 fun panda facts for a Y5 homework project. One sentence each.
This uses way fewer tokens. It says exactly what you want (5 facts), who it's for (Y5 homework), and what shape (one sentence each). The AI now has all it needs, and no waste.
In pairs, give pupils this long-winded message and challenge them to cut it in half without losing what they want.
> Hi AI, I hope you are well. Would you please be able to write me a short story, maybe about a page long, about a friendly dragon who goes to school, if that is OK with you? Thank you very much.
Read a couple of the best rewrites out loud. Which is the shortest that still gets a good story? The class will spot that the polite bits and the 'if that is OK' bits can go, but the important bits (dragon, school, page long) have to stay.