Guide · Tokens & costs
AI tokens explained: what a token is, how it is counted, and what it costs
AI tokens are the chunks of text a language model reads and writes: whole words, pieces of words, punctuation and spaces. In English one token averages about four characters, or three quarters of a word. Each model’s tokenizer splits text its own way, and every API price and context limit is counted in tokens.
By Tahir NazirUpdated 10 min read
On this page
- What is a token in AI? A worked example
- How are AI tokens calculated?
- How many tokens is a word?
- Input tokens vs output tokens: why output costs more
- Why other languages use more tokens
- Why does Claude count my text differently from GPT?
- How do tokens affect cost and the context window?
- How to use fewer tokens
- Questions people ask
What is a token in AI? A worked example
A token is the smallest unit of text a language model works with. Language models don’t see letters or words. Before your text reaches the model, a tokenizer cuts it into pieces from a fixed vocabulary, typically 100,000 to 260,000 entries, and replaces each piece with a number. The model reads that list of numbers and predicts the next number, one at a time. Those pieces are tokens.
Common words get a token of their own. Rarer words are built from several pieces, and a space is usually glued to the front of the word that follows it. Each token also has an ID, its position in the vocabulary, and that number is all the model actually sees:
Tokenizers split unbelievable words in 2026.
- Token4421
- izers24223
- ␣split12648
- ␣unbelievable83614
- ␣words6391
- ␣in306
- ␣220
- 2021323
- 621
- .13
10 tokens · o200k_base · ␣ = space
A few more inputs show the same patterns. Each chip in the last column is one token:
| Text | Tokens | Pieces |
|---|---|---|
| The quick brown fox jumps over the lazy dog. | 10 | The ␣quick ␣brown ␣fox ␣jumps ␣over ␣the ␣lazy ␣dog . |
| Tokenization is weird. | 5 | Token ization ␣is ␣weird . |
| unbelievably | 3 | un bel ievably |
| ChatGPT | 2 | Chat GPT |
| 2026-10-08 | 6 | 202 6 - 10 - 08 |
| 1234567 | 3 | 123 456 7 |
␣ marks a space that belongs to the token. Counts from gpt-tokenizer 4.0, o200k_base (GPT-4o to GPT-5.x).
Three things stand out. A token isn’t a word: “Tokenization” is two tokens, and a date is six. The space matters: “unbelievably” on its own is three tokens, but “␣unbelievably” after a space is a single token, because that’s how it usually appears in text. And numbers are split into groups of up to three digits, which is one reason models can stumble on long arithmetic.
These tokens have nothing to do with crypto tokens. In AI, a token is only a piece of text, and providers bill for them per million. (For a one-line definition, see the token glossary entry.)
How are AI tokens calculated?
A program called a tokenizer calculates them by matching your text against a fixed vocabulary. It splits the text into pieces the vocabulary contains, turns each piece into its ID, and the number of IDs is your token count. Nothing is estimated: the same text on the same tokenizer always gives the same tokens.
Most current tokenizers use byte pair encoding (BPE). OpenAI’s tiktoken documentation sums up what it does: it is reversible and lossless, so tokens convert back to exactly the original text; it compresses the text, so there are fewer tokens than bytes; and it lets the model see common subwords. The vocabulary is learned from training text, which is why common English words get a token of their own and rare words are built from pieces.
That is also why there is no universal formula from words to tokens. Each model family trained its own vocabulary, so the same sentence gives different counts on GPT, Claude and Gemini. Rules of thumb get you close for English prose; for anything else, run the real tokenizer, as our guide to counting tokens in Python and JavaScript shows.
How many tokens is a word?
OpenAI’s own rule of thumb is that one token is about four characters or three quarters of a word of English. Our measurement agrees: 1,304 words of ordinary English prose came to 1,628 tokens with o200k_base, which is 0.80 words or 4.8 characters per token. Useful conversions for English:
- 100 tokens ≈ 75 words ≈ a short paragraph
- 1,000 tokens ≈ 750 words ≈ a short blog post
- 100,000 tokens ≈ 75,000 words ≈ a full novel
These are averages for prose. Code, JSON, tables and logs contain lots of punctuation and short symbols, so they use more tokens per word: the same tokenizer needed about 2.1 tokens per whitespace-separated word on a TypeScript file and 3 on a JSON file. The tokens to words converter has measured ratios for each tokenizer, for languages from Spanish to Swahili, and for code, Markdown and JSON.
Input tokens vs output tokens: why output costs more
Input tokens are everything you send (the system prompt, the conversation so far, documents and your question); output tokens are everything the model writes back. APIs price them separately, and output is the dearer side: on the popular models in our data, output costs 3× to 5× the input price (prices as of 2026-11-12).
The reason is how the work is done. The model reads your whole prompt in one parallel pass, but writes its reply one token at a time, and each new token needs another pass through the model. Reasoning models add hidden “thinking” tokens before the answer, and those are billed as output too. Our guide to estimating LLM API costs turns both sides into a monthly bill.
Images and audio become tokens as well. Their count depends on the image size or the length of the audio and on each provider’s own rules, not on any text, so check the usage figures a real request returns.
Why other languages use more tokens
Tokenizer vocabularies are built from training text, and most of that text is English. Languages that appear less often, or use other scripts, get fewer whole-word tokens, so the same meaning costs more. Newer tokenizers have narrowed the gap a lot:
| Text | o200k_base (GPT-4o to GPT-5.x) | cl100k_base (GPT-4, GPT-3.5) |
|---|---|---|
| Das ist ein Beispiel. (German) | 5 | 5 |
| यह एक उदाहरण है। (Hindi) | 5 | 20 |
| 这是一个例子。 (Chinese) | 5 | 6 |
| مرحبا بالعالم (Arabic) | 4 | 10 |
The German, Hindi and Chinese sentences all mean “This is an example.”; the Arabic one means “Hello, world”. The Hindi sentence took four times as many tokens on the older tokenizer.
- German
- 5 tokens with o200k5 tokens with cl100k
- Hindi
- 5 tokens with o200k20 tokens with cl100k
- Chinese
- 5 tokens with o200k6 tokens with cl100k
- Arabic
- 4 tokens with o200k10 tokens with cl100k
So if you work in Hindi, Arabic, Urdu or similar languages, the tokenizer generation matters as much as the price per token. Always measure with your own text.
Why does Claude count my text differently from GPT?
Because every model family has its own tokenizer, the same text gives different counts on GPT, Claude, Gemini, Llama or Qwen. A price per million tokens is only comparable once you know how many tokens your text becomes on each model.
- OpenAI publishes the tokenizers for its models up to GPT-5.x in tiktoken, so those counts can be exact offline.
- Open-weight models (Llama, Qwen, Mistral, DeepSeek) ship their tokenizer with the weights, so counts can be exact too.
- Anthropic doesn’t publish Claude’s tokenizer, so any offline number for Claude is an estimate. Its free
count_tokensendpoint gives the official count, which Anthropic says can differ slightly from a real request. - Google offers a free
countTokensendpoint for Gemini, and its Python SDK (google-genai) can also count text locally for supported Gemini models with itsLocalTokenizer.
Counts can shift even within one family: Anthropic says Claude Opus 4.7 and later use a newer tokenizer that turns the same text into about 30% more tokens than earlier Claude models. That’s why our token counter doesn’t guess for Claude and Gemini: it shows OpenAI’s count marked for reference only, and fetches the official count when you add your own key.
Free toolAI token counterExact counts for models with public tokenizers, official Claude and Gemini counts with your key, and a view that shows every token.How do tokens affect cost and the context window?
Tokens set both what a request costs and whether it fits:
- Cost. APIs charge per million input tokens and per million output tokens, and output is usually several times more expensive. The LLM cost calculator turns token counts into a monthly bill.
- Context. A model’s context window is the most tokens it can handle in one request: your prompt, the conversation so far, any documents, and the reply. Context windows explained covers what counts, and the context window checker tells you whether a document fits.
In a chat or agent, the whole conversation is sent again on every turn. A 20-turn conversation therefore pays for the first message 20 times, which is why long sessions get expensive and why prompt caching exists.
How to use fewer tokens
- Send only what the model needs: trim logs, strip boilerplate, and summarise long histories instead of resending them.
- Minify JSON you send as input; whitespace and indentation are tokens too.
- Ask for short output when that’s all you need. Output tokens cost the most.
- Use prompt caching for long, repeated prefixes like system prompts and documents.
- Pick a newer tokenizer for non-English text when you have the choice.
FAQ
Questions people ask
How many tokens is 1,000 words?
About 1,250 tokens for ordinary English prose with OpenAI’s o200k_base tokenizer (we measured 1.25 tokens per word; OpenAI’s rule of thumb gives about 1,330). Code, JSON and non-English text use more. Paste your text into the token counter for the exact number.
Is a token the same as a word?
No. Short common words are usually one token, but longer or rarer words are split into pieces, punctuation is usually its own token, and the space before a word is normally part of that word’s token. Runs of spaces, tabs and blank lines become tokens too, so indented code and pretty-printed JSON use noticeably more tokens than minified text.
How much is 1,000 tokens?
In text, about 900 English words on GPT’s o200k_base tokenizer (measured on the English text of the Universal Declaration of Human Rights) or about 600 on current Claude models (Anthropic’s figure). In money, prices are quoted per million tokens, so 1,000 tokens costs a thousandth of the listed rate: on Claude Sonnet 5.5, $0.002 as input or $0.01 as output (prices as of 2026-11-12).
How many pages are 10,000 tokens?
About 11 to 17 single-spaced pages of English at 500 words a page. That is roughly 5,600 words on current Claude models (Anthropic’s figure) and 8,700 on GPT’s o200k_base tokenizer (measured on the English Universal Declaration of Human Rights). Code, tables and other languages fit fewer pages. The tokens to words converter does this sum for other tokenizers and languages.
Why does Claude count my text differently from GPT?
Each model family has its own tokenizer with a different vocabulary. Anthropic doesn’t publish Claude’s tokenizer, so official Claude counts come only from its count_tokens endpoint, which is free to call with an API key. Claude models differ among themselves too: Anthropic says Opus 4.7 and later count about 30% more tokens for the same text.
Are AI tokens the same as crypto tokens?
No. In language models a token is a piece of text (a word, part of a word or a punctuation mark) that the model reads or writes, and API providers bill per million of them. Crypto tokens are digital assets on a blockchain. The two only share a name: an AI token isn’t something you can buy, hold or trade on its own.
Who pays for AI tokens?
Whoever owns the API key. With an API, every input and output token is billed to the account that made the request, at that model’s price per million tokens. Subscription apps such as ChatGPT and Claude plans charge a flat monthly fee with usage limits instead, so you don’t see a per-token bill. The LLM cost calculator prices API usage.
Do images count as tokens?
Yes. Vision models convert images to tokens too, and the count depends on the image size and the provider’s rules rather than on any text. A 1920 × 1080 screenshot is about 2,691 tokens on current Claude models; our guide to how many tokens an image is gives each provider’s formula.
Try it
Tools from this guide
- AI Token CounterCount tokens for GPT, Claude, Gemini, DeepSeek, Qwen and more.
- Tokens to Words ConverterConvert between tokens, words and pages.
- LLM API Cost CalculatorEstimate per-request, daily and monthly API costs.
- Context Window CheckerSee whether your text fits each model’s context window.
- Prompt DiffCompare two prompt versions side by side.
Glossary
Key terms in this guide
Keep reading
Related guides
- Tokens & costsHow to calculate LLM API costs: the formula, worked examples and the traps12 min read
- Tokens & costsContext windows explained: what counts, what happens at the limit, and how to check fit12 min read
- Tokens & costsPrompt caching explained: how it works on OpenAI, Anthropic and Gemini, and when it pays13 min read