Tokens: the invisible unit running your AI bill
Why tokens quietly drive cost, context, latency, and correctness at work
The invoice that quietly tripled
Open the OpenAI Tokenizer in a second tab: [platform.openai.com/tokenizer](https://platform.openai.com/tokenizer). It's free, needs no sign-in, and gives you the same view your AI provider is billing off. Paste any of the three snippets below and watch the text split into coloured chunks. Every chunk is one token, and the number at the bottom is exactly what you'd be charged for.
You bolted an AI feature onto your team's workflow. Perhaps a summariser sitting on top of customer emails, or a helper that drafts first-pass replies, or an internal chatbot that fields the questions HR keeps getting. It shipped, people liked it, everyone moved on. Then the invoice from your AI provider landed at the end of the month, and it was roughly three times what you'd budgeted, and nobody on the team could quite explain why.
Welcome to tokens. Tokens are the unit your AI provider bills you in, and if nobody on the team really understands them, that surprise invoice tends to become a monthly ritual. Cost is only half the story, though. Tokens also set the ceiling on how much context your AI can hold in one conversation, how quickly it responds under load, and even which questions it can reliably answer at all. There's a well-known example where a state-of-the-art model can't consistently tell you how many Rs are in the word 'strawberry'. That's tokens too.
Over the next fifty minutes we'll open tokens up, watch them in a real tokeniser, and finish with a set of habits worth taking back to work. Some will save you real money. All of them will help you spec, buy, and manage AI more confidently.
Hi team, quick update on the Q3 project. We've locked the timeline, the budget is signed off, and the client meeting is confirmed for next Thursday at ten. Let me know if anything needs a second look before then. Thanks.
Cześć zespół, krótka aktualizacja projektu Q3. Zablokowaliśmy harmonogram, budżet jest zatwierdzony, a spotkanie z klientem potwierdzone na przyszły czwartek o dziesiątej. Dajcie znać, jeśli coś wymaga jeszcze przeglądu. Dzięki.
The Party of the First Part hereby indemnifies, defends, and holds harmless the Party of the Second Part from any and all liabilities arising under Section 4.2(b)(iii) of the Master Services Agreement dated 12 March 2024.
Compare the three token counts and ask the team:
- Which of these three uses the most tokens per unit of information? Any surprises in the ranking?
- What happens to the long compound words and the section reference in the legal snippet?
- If you were paying by the token (which you are, on every commercial AI API), which of these three would be the most expensive per idea communicated?
Copy a real message from your team's Slack, an email you sent this morning, and, if you can, the same content in a language your team uses with customers (Polish, French, Arabic, Chinese, whichever is relevant). Watch the token counts shift. The point is not academic: if your team supports customers in five languages, some of those languages are quietly costing your budget roughly twice as much per query as the English equivalent.
What is a token, actually?
A modern AI language model doesn't read your prompt as a string of letters, and it doesn't quite read it as words either. Sitting between the two is a token.
A token is a chunk of text that the model has decided is worth treating as one unit. For common English words that works out at roughly one token per word, plus a token or two for punctuation and spacing. Less common words get split into pieces (the word 'tokeniser' itself is three tokens: `token`, `is`, `er`). Names, code, jargon, non-English text, emoji, and made-up words all balloon the count, because every unfamiliar fragment needs its own separate token.
Every input you send and every output the model generates is measured in tokens. Every commercial provider prices you in tokens. Every model has a hard token ceiling per conversation. This is the unit that quietly runs everything, and once you can see it, a lot of AI's day-to-day behaviour stops being mysterious.
Take a paragraph from a policy document, product spec, or customer FAQ that your team actually uses. Paste it into the tokeniser. Then try three variations: the original, the same paragraph in ALL CAPS, and the same paragraph translated to another language via any translator. Ask the team to predict which version will use the most tokens before you press Show. The pattern of surprise you'll get is exactly the pattern that shows up in every AI invoice.
Three places tokens quietly bite
You might file tokens under 'developer concern' and move on. It's a mistake. Tokens shape three separate parts of the day-to-day AI experience, and each one lands squarely in a business context: your budget, your product's memory, and your users' patience.
Before you tap the cards below, take a guess: in your own team's use of AI, where might tokens quietly be costing you today?
Tap each card to reveal.
Cost per API call
Every commercial AI provider prices you by the token, with different rates for input and output, and different rates by model. A single 4,000-token conversation with a mid-range model costs a few pence. Multiply that by ten thousand customer queries a month and you're into serious money quickly. Prompt hygiene is a budget line, not an aesthetic preference.
The context window
Every model has a hard maximum number of tokens it can hold in one conversation, called the context window. Once the conversation exceeds it, older content gets silently dropped. That's why a chatbot fed a whole employee handbook can 'forget' the first chapters by question three, and why very long support-chat threads suddenly lose track of the customer's original problem.
Latency plus energy
More tokens means more compute per response, which means slower answers, higher server load, and more electricity per query. A user-facing feature that has to respond in under a second lives or dies by its token count. And from a sustainability angle, a lean 500-token prompt is a materially greener choice than a bloated 5,000-token one for the same answer.
A well-shaped prompt costs less, keeps more of its memory, and lands on the user faster. Three business wins for the price of one habit.
Give each pair one of these real-feel situations and ask which of the three token pressures is dominant.
- Your ops team's monthly OpenAI bill triples the month after a new feature ships that lets users paste in whole PDFs.
- A customer-service bot starts contradicting itself in long chats, giving the customer the wrong policy on their fifth question.
- A live sales assistant takes ten seconds to reply, which is enough for the customer to alt-tab away and forget they were talking to anyone.
Good answers name the dominant pressure. Excellent answers spot that the three often overlap in production.
Why AIs can't count letters
Back to strawberry. The reason a modern model sometimes confidently reports two Rs isn't lack of intelligence, and it certainly isn't malice. It's tokens.
When you type strawberry, the tokeniser doesn't hand the model ten letters. It hands it something like `straw` + `berry`, two chunks. The model then dutifully counts the Rs it can see in those chunks and gives you an answer. It isn't being lazy. It's answering the question it was actually shown, which isn't the question you meant to ask.
The same shape underlies its trouble with anagrams, with reversing words character by character, with counting emoji, with formatting dates on odd token boundaries, and with any task where the meaningful unit is a letter or a symbol. In a business context this matters most when you're picking a tool: character-level jobs (spell-checking legal-code identifiers, validating serial numbers, sorting an alphabetical list precisely) aren't the natural strength of a language model, and knowing why saves you from procuring the wrong thing.
Give the team three challenges and have them predict which the AI will get right and which it'll bungle. Then try live in whichever model you use at work.
- How many letters are in the word 'onomatopoeia'?
- Spell 'lasagna' backwards, one letter per line.
- How many emoji appear in this sentence: 'I love 🍕 and 🥞 and 🍰 and 🍿'?
All three are letter-level or symbol-level tasks. The pattern to walk away with: token-friendly questions get token-friendly answers, and letter-friendly questions do not. If your workflow depends on the second kind, pair the AI with a small deterministic script that handles the letter work.
The multilingual tax
If your business operates in more than one language, tokens matter to you specifically. English is, roughly, the cheapest major language in the world to run through a modern AI, because the training data was dominated by English and the tokeniser learned to compress it efficiently. Everything else costs more per idea, sometimes considerably more.
The reason is simple. For a language the tokeniser didn't see as often, common words get broken into more, smaller pieces. A friendly Polish greeting costs roughly twice the tokens of the same greeting in English. A Chinese, Japanese, or Arabic sentence can cost two to three times as much again. Translate your customer-support workflow into five languages and you haven't only multiplied the translation work; you've quietly multiplied your API bill in a ratio that varies by language.
Matching the language to its rough token weight is a useful first pass before you commit to any multilingual rollout.
Take one customer-facing message your team sends regularly (a shipping-update template, an onboarding email, a policy explainer). Ask the team to guess: if you translate this into five languages your customers use, roughly what happens to the per-message API cost? Then run each translation through the tokeniser and see. The gap between the guess and the reality is usually the shape of a decision that hasn't been made yet.
Bloated prompt, tight prompt
Every token in your prompt costs money, eats context, and slows the response. Every token in the model's output does the same. The game, then, is to get the same result in fewer tokens.
The two prompts below ask for exactly the same thing. Both work. One is roughly four times cheaper than the other. Copy each into the tokeniser to see the difference for yourself, and remember that this multiplier applies to every single call your product makes.
Hello, I hope you are having a wonderful day today. I was wondering if you would be so kind as to please generate for me, if it's not too much trouble, a fairly short list of maybe about five or so of the key things that a new employee at a UK small business might genuinely want to know about their first week on the job, ideally in a format that is easy to read and reasonably clear. Thank you very much in advance for your help!
Roughly 90 tokens, most of them padding. Chatty framing costs the same as substantive content, and the model tends to mirror the tone back at length in the response. On a paid API that's roughly 3 to 4 times the price of the tight version, and every request pays it.
List 5 things a new UK small-business employee should know about their first week. Bullet points, one line each, no preamble.
Roughly 25 tokens. Names the audience, sets the count, locks the format, kills the waffle. Same output shape, a fraction of the cost, and the model tends to be MORE useful because the constraints are unambiguous.
In pairs, hand your team this bloated system prompt and challenge them to cut it to under 30 tokens (roughly 20 words). No losing intent, no losing brand tone. Test the winners in a real AI at the front and see whether the answers get better as the prompt gets tighter.
> Please could you be so kind as to write, when you have a moment, a fairly detailed and thoughtful summary, of maybe about a paragraph or two in length, of the main safety-training points that a new warehouse operative might want to know before their first shift, in a warm and welcoming way, if that would be alright with you?
The game rewards deleting politeness padding, hedging, and any redundant instructions. Adults new to prompting are often surprised how much tighter the output gets when the input tightens.
The context window and why 'just paste the whole handbook' rarely works
Once a team gets comfortable with AI, one idea comes up almost every time: 'Let's just paste our whole handbook / product catalogue / support archive / codebase into the system prompt so the AI always knows the answer.'
It sounds reasonable, and it isn't unreasonable, but the context window turns it into a design decision rather than a default. Even a very generous 128,000-token window sounds huge until you notice that a typical employee handbook, plus a customer's full order history, plus the conversation so far can easily blow through it. And even before you hit the ceiling, models tend to pay less attention to content buried deep in a long prompt. The phenomenon is called 'lost in the middle', and it's why an important policy on page forty of a long handbook can quietly get ignored.
The pragmatic pattern most teams end up using is called retrieval-augmented generation, or RAG: keep the handbook in a searchable store, fetch only the two or three most relevant passages for the current question, and pass just those into the prompt. Fewer tokens, cheaper, faster, and the model actually reads the bit that matters.
Have the team pick one document-heavy AI feature you'd realistically build (an HR helper, a product-support bot, a policy-lookup for a sales team). Ask them to sketch two versions on the whiteboard: one where the whole document gets pasted into every prompt, and one where a RAG lookup fetches only the relevant paragraphs. For each version, get them to name the per-call cost, the failure mode, and what they'd need to build. The exercise is a good primer for anyone about to scope a real AI project.
So what would you do?
Time to bring everything together. You've been asked to spec a small AI feature at work: a first-pass summariser for customer-support tickets, so agents can triage faster. You have a modest budget of about £300 per month, roughly 2,000 tickets a month across three languages (English, Polish, Arabic), and stakeholders who want responses fast enough to feel instant in the agent's dashboard.
Rough maths: £300 across 2,000 tickets is 15p per ticket for all input and output combined, before overhead. On today's mid-range API rates that's very achievable in English, tighter in Polish, and easily over budget in Arabic unless you plan for it. Your first design decision matters.
Split the team into groups of three. Each group has the same brief: design the summariser feature to hit the £300 budget across all three languages. Ten minutes. Watch for the good decisions: a tight system prompt with a hard word cap on the summary, a smaller and cheaper model for the first pass (with escalation to a larger model only on flagged tickets), input truncation, careful thought about which languages get English-first machine translation, and RAG only where the tickets need policy context. Groups that finish early can budget-model their design in a spreadsheet.