Tokens: how AIs read your words
Why AIs can't count the Rs in 'strawberry', and what that tells us
Ask ChatGPT how many Rs are in 'strawberry'
Before we say anything else, let's watch tokens happen live.
Open a second tab and go to platform.openai.com/tokenizer. It's free, no sign-in needed. Whatever you paste in, it splits into coloured chunks and tells you how many tokens the AI would see.
Try one of the prompts below and watch the colours change.
Try it. Type: How many letter Rs are in the word strawberry? into any chatbot you like.
There's a really good chance it'll say two. It isn't trying to trick you and it isn't being lazy. The AI honestly cannot see the word the way you and I do.
Over the next half hour we'll work out why. It turns out the AI doesn't read your writing letter by letter, or even word by word. It reads in chunks called tokens, and once you know how tokens work, a lot of the AI's weird behaviour suddenly makes sense: why it forgets what you said earlier in a long chat, why the same idea in Polish or Urdu 'costs' more, and yes, why it can't count the letters in a fruit.
The football match on Saturday was really good, especially the last ten minutes.
Sobotni mecz piłkarski był naprawdę dobry, zwłaszcza ostatnie dziesięć minut.
That skibidi lesson was actually kinda peak 😂 #vibes
Look at the three token counts together and ask the class:
- Which one uses the most tokens? Which uses the fewest? Did that match your guess?
- What happens to emoji, hashtags and slang? Are they cheap or expensive?
- If the AI charges by the token (which it usually does), which one of these would cost the most to send?
So what actually is a token?
When you send a message to an AI, it doesn't line up your text letter by letter. It also doesn't read it word by word. Somewhere in between sits the token.
A token is a chunk of text that the AI has decided is worth treating as a single piece. For common English words that's usually one whole word per token, plus a token for the space or punctuation. For rarer words the tokeniser chops them into smaller bits. The word strawberry, for example, gets split into something like `straw` + `berry`, two chunks instead of one.
Every message you send AND every reply the AI gives back is counted in tokens. Not letters. Not words. Tokens are the unit the AI actually pays attention to.
Three ways tokens quietly matter
Tokens sound like something only coders would worry about. They're not. They quietly shape a lot of what you see when you use any AI.
Before you tap the cards below, have a guess. Where do you think a token count might secretly be causing a problem in your everyday AI use?
Tap each card to reveal.
Cost
Every commercial AI charges by the token, with different prices for what you send in and what it sends back. That's usually invisible to you on free ChatGPT, but it's why 'AI credits' run out on paid apps and why long chats eat them faster than short ones.
Memory (the context window)
Every AI has a maximum number of tokens it can hold in a single conversation. Once you hit that limit, the oldest parts of the chat drop off the back. That's why a really long ChatGPT thread suddenly starts forgetting things you told it near the start.
Speed
More tokens means more work for the AI, which means a slower reply. A tight 200-token question is much faster to answer than a bloated 2,000-token one. If your AI feels sluggish, the length of the conversation is often the reason.
So keeping your messages tight isn't just polite. It saves money, it stops the AI forgetting things, and it makes replies faster. Three wins from one habit.
Why AIs can't count letters
Back to the strawberry problem. The reason the AI gets it wrong isn't that it's stupid, and it definitely isn't being cheeky. It's tokens.
When you type strawberry, the tokeniser doesn't hand the AI ten letters. It hands it two chunks, roughly `straw` and `berry`. The AI then counts the Rs in what it can actually see, which is those chunks. It answers the question it was really shown, which isn't quite the question you meant to ask.
Same reason it fumbles anagrams, gets confused if you ask it to spell a word backwards one letter at a time, or miscounts emoji in a sentence. Anything where the letter (or the individual symbol) is the important thing is a task tokens quietly wreck.
Which words are cheap, which are pricey?
Not all words cost the same in tokens. The tokeniser was trained on a mountain of text (mostly English, mostly from the internet), so it has neat little chunks ready for anything common. Anything unusual has to be built up from smaller pieces.
Have a quick guess before you look at the match below: which do you think uses more tokens, the word the or the word antidisestablishmentarianism? And what happens if you type your friend's name into the tokeniser?
Bad prompt, good prompt
Every extra token in your message costs money, chews up memory and slows the reply. Every extra token in the AI's reply does the same. So the game is simple: get the same result in fewer tokens.
The two prompts below both ask for exactly the same thing. Both work. But one is roughly three times cheaper to send, and often gives a better answer because the instructions are clearer. Paste each one into the tokeniser and see the count for yourself.
Hi there, I hope you're doing OK today. I was just wondering, if it's not too much trouble, whether you might be able to please give me maybe five or so quick tips or ideas for how a Year 8 student in the UK might revise for their end of year Geography test, ideally in a way that's fun and not too boring. Thanks so much!
Roughly 80 tokens, most of which is padding. All the 'please' and 'if it's not too much trouble' and 'thanks so much' costs the same as the useful part. The AI often mirrors your chatty style back at length too, which makes the reply expensive as well.
5 revision tips for a UK Y8 Geography end-of-year test. Bullet points, one line each, no waffle.
Roughly 25 tokens. It names the audience (Y8, UK), the subject, the count, the format, and it tells the AI not to pad the answer. Same job, a third of the cost, and usually a sharper reply.
So what would you do?
Time to put everything together. Imagine you're building a small AI helper for a school project. It's supposed to answer quick questions about the plot of Romeo and Juliet for younger pupils.
You've been given a free-tier account with 30,000 tokens a day and you're expecting about 100 questions. Do some quick maths: that's 300 tokens per question, split between what you send AND what the AI sends back.
On your first test, you get: 'How many Ss are in the name Rosencrantz?' from a pupil. The AI says 3. You count. It's 2. And your bloated system prompt has already used up half your budget for that message.
You need a plan.