Tokens: how AIs read text
Why the tiny chunks matter for cost, context, and correctness
Why 'strawberry' breaks AIs
Open the OpenAI Tokenizer in a second tab: [platform.openai.com/tokenizer](https://platform.openai.com/tokenizer). It's free, no sign-in needed.
Copy one of the three prompts below into it, and watch how it splits your text into coloured chunks. Each chunk is a token. The number at the bottom is what the AI actually sees and gets charged for.
Ask ChatGPT how many letter Rs are in the word strawberry. There's a decent chance it'll say two. It won't be trying to trick you. It genuinely doesn't see the word the way you and I do.
AIs don't read letter by letter. They read in bigger chunks called tokens, and once you know how tokens work, a lot of AI's weird behaviour starts to make sense: why long conversations suddenly get expensive, why the same idea in Polish costs three times as much to process, why a chatbot forgets what you said twenty minutes ago, and yes, why it can't count the Rs.
In the next thirty-five minutes we'll take tokens apart, see them live in a real tokeniser, and finish with a habit-set that could easily save you a few pounds a month on any AI project you build.
The quick brown fox jumps over the lazy dog on a summer afternoon.
Szybki brązowy lis przeskakuje leniwego psa w letnie popołudnie.
The zorglemat pinged at 8:47am 🎉 because MDMA-blockchain-quantum said so #vibes.
Compare the three token counts and ask the class:
- Which of these three uses the most tokens? Why do you think that is?
- What happens to unusual or made-up words? Emoji? Numbers?
- If you were charged by the token (which you actually are, on most AI APIs), which of these prompts would be the most expensive per idea communicated?
What is a token, actually?
The AI doesn't read your prompt as a string of letters. It doesn't even read it as words. Somewhere between the two sits the token.
A token is a chunk of text that the AI has decided is worth treating as one unit. For common English words that ends up being roughly one token per word, plus a token or two for punctuation and spacing. For less common words the tokeniser breaks them into pieces ("tokeniser" itself is three tokens: `token`, `is`, `er`). For non-English text, for emoji, and for made-up words, the token count balloons because each rare piece needs its own separate token.
Every input you send and every output the AI generates is measured, priced, and length-limited in tokens. Not letters, not words. Tokens.
Three places tokens quietly bite
You might think tokens are a developer's problem. They're not. They quietly shape a lot of everyday AI behaviour.
Before you tap the cards below, have a think: where in your day-to-day AI use might a token count secretly be causing you a problem?
Tap each card to reveal.
Cost per API call
Every commercial AI API charges by the token, with different rates for input and output. A 4,000-token conversation with GPT-4o costs roughly 3-4 pence; do that thousands of times a month and it adds up fast. Prompt hygiene isn't an aesthetic choice, it's a budget line.
The context window
Every model has a maximum token limit for a single conversation (its "context window"). Once you hit it, the AI starts forgetting the oldest parts of the conversation. That's why long ChatGPT threads suddenly seem to lose the plot: they literally have.
Latency + energy
More tokens means more compute, which means slower responses AND more energy per query. A 500-token prompt is faster and greener than a 5,000-token one, even if the answer is the same. This connects tokens to the AI Energy lesson elsewhere on the Lab.
So a well-shaped prompt is cheaper, keeps its memory longer, and hits the planet less hard. Three wins for the price of one habit.
Why AIs can't count letters
Back to the strawberry problem. The reason the AI gets it wrong isn't lack of intelligence, and it definitely isn't malice. It's tokens.
When you type strawberry, the tokeniser doesn't hand the AI ten letters. It hands it something like `straw` + `berry`, which is two chunks. The AI then dutifully counts the Rs it can see in those chunks and reports back. It's not being lazy; it's answering the question it was actually shown, which isn't the question you meant to ask.
Same reason it struggles with anagrams, with reversing words character by character, with counting emoji, with formatting dates that fall on unusual token boundaries. Any task that treats a letter as the unit of meaning is a task tokens quietly sabotage.
Bad prompt, good prompt
Every token in your prompt costs money, eats context, and slows the response. Every token in the AI's output does the same. So the game is: get the same result in fewer tokens.
The two prompts below ask for exactly the same thing. Both work. One is roughly 4x cheaper than the other. Copy each into the tokeniser to see the difference for yourself.
Hello, I hope you're well today. I was wondering if you would be so kind as to please generate for me, if it's not too much trouble, a fairly short list of maybe about five or so key things that a Year 11 student in the UK might genuinely want to know about the topic of climate change, ideally in a format that is easy to read and reasonably clear. Thank you very much in advance!
~90 tokens, most of them padding. Chatty framing costs the same as substantive content, and the AI often mirrors it back at length. On a paid API this is roughly 3-4x the price of the tight version.
List 5 climate-change facts a UK Y11 student should know. Bullet points, one line each, no preamble.
~25 tokens. Names the audience, the count, the format, and forbids waffle. Same output shape, fraction of the cost. The AI also tends to be MORE useful because the constraints are clearer.
So what would you do?
Time to put it into practice. You're a Y13 building a small AI chatbot for a business studies project: it'll answer questions about your fictional coffee shop. You've got a free-tier API key with 40,000 tokens per day, and you're expecting about 200 questions.
Do some quick maths: 40,000 tokens ÷ 200 questions = 200 tokens per question total, split between prompt AND response. That's tight. A greeting + a menu list + a polite sign-off could easily blow it in one exchange.
You need a plan.