How AIs read your words

Let's open a magic window that shows what an AI sees. Go to [platform.openai.com/tokenizer](https://platform.openai.com/tokenizer) in a new tab. It's free and you don't need to sign in.

Paste one of the prompts below into the box. The tool will colour the text in chunks. Each coloured chunk is called a token, and that's what the AI treats as one piece.

Before you paste each one in, have a quick guess at how many chunks it will turn into.

Here's a game. Ask any AI chatbot: how many letter Rs are in the word 'strawberry'? There's a good chance it will say two. But the real answer is three. Try it yourself and see.

The AI isn't trying to trick you, and it isn't broken. It just doesn't read words the same way you do. When you read 'strawberry', you see ten letters in a row. When the AI reads 'strawberry', it sees something quite different.

Over the next fifteen minutes we're going to find out what the AI actually sees, why it matters, and one clever trick to talk to an AI in a way it understands.

When you type a message into an AI, you probably think it reads it letter by letter. Actually it doesn't. It reads in chunks, and each chunk is called a token.

For common English words that the AI has seen a million times, a token is often just the whole word. So 'cat' is one token, and so is 'dog', and so is 'homework'. Easy.

But for tricky words, made-up words, or words in other languages, the AI has to break them into little pieces. Something like 'skibidi' might turn into three or four tokens because the AI has never really learned it as one thing. That's the whole idea. Tokens are the pieces the AI uses to read.

Tokens might sound like a computer-nerd thing. Actually they change how the AI feels to use.

Before you tap the cards below, have a quick guess. Why might it be a good idea to use fewer tokens when you talk to an AI?

So a short, clear message is cheaper, faster, AND easier for the AI to remember. Three good things for the price of one habit.

Back to the strawberry trick from the start. Why does the AI keep saying there are two Rs?

Here's the answer. When you type 'strawberry', the AI doesn't see s-t-r-a-w-b-e-r-r-y. It sees two chunks: 'straw' and 'berry'. That's all it has to look at. So when you ask it to count the Rs, it counts the Rs inside those two chunks, and it misses one.

This is why AIs are often rubbish at things where letters really matter. Counting letters, spelling words backwards, working out anagrams, that kind of thing. The AI is amazing at some things and quite bad at others, and knowing about tokens helps you spot which is which.

Now for the useful bit. If tokens cost money AND fill up the AI's memory AND slow it down, then a shorter message is a better message. But short doesn't mean rude. It means clear.

The two prompts below are trying to do the same thing. Both work. But one uses about four times as many tokens as the other. Look at them and see if you can spot why.