What is a token in AI? A plain-English guide with real examples

What a token is#

Language models don’t see letters or words. Before your text reaches the model, a tokenizer cuts it into pieces from a fixed vocabulary, typically 100,000 to 260,000 entries, and replaces each piece with a number. The model reads that list of numbers and predicts the next number, one at a time. Those pieces are tokens.

Common words get a token of their own. Rarer words are built from several pieces, and a space is usually glued to the front of the word that follows it. Each token also has an ID, its position in the vocabulary, and that number is all the model actually sees:

A few more inputs show the same patterns. Each chip in the last column is one token:

Three things stand out. A token isn’t a word: “Tokenization” is two tokens, and a date is six. The space matters: “unbelievably” on its own is three tokens, but “␣unbelievably” after a space is a single token, because that’s how it usually appears in text. And numbers are split into groups of up to three digits, which is one reason models can stumble on long arithmetic.

How many tokens is a word?#

OpenAI’s own rule of thumb is that one token is about four characters or three quarters of a word of English. Our measurement agrees: 1,304 words of ordinary English prose came to 1,628 tokens with o200k_base, which is 0.80 words or 4.8 characters per token. Useful conversions for English:

• 100 tokens ≈ 75 words ≈ a short paragraph
• 1,000 tokens ≈ 750 words ≈ a short blog post
• 100,000 tokens ≈ 75,000 words ≈ a full novel

These are averages for prose. Code, JSON, tables and logs contain lots of punctuation and short symbols, so they use more tokens per word: the same tokenizer needed about 2.1 tokens per whitespace-separated word on a TypeScript file and 3 on a JSON file.

Why other languages use more tokens#

Tokenizer vocabularies are built from training text, and most of that text is English. Languages that appear less often, or use other scripts, get fewer whole-word tokens, so the same meaning costs more. Newer tokenizers have narrowed the gap a lot:

So if you work in Hindi, Arabic, Urdu or similar languages, the tokenizer generation matters as much as the price per token. Always measure with your own text.

Different models, different counts#

Every model family has its own tokenizer, so the same text gives different counts on GPT, Claude, Gemini, Llama or Qwen. A price per million tokens is only comparable once you know how many tokens your text becomes on each model.

• OpenAI publishes the tokenizers for its models up to GPT-5.x in tiktoken, so those counts can be exact offline.
• Open-weight models (Llama, Qwen, Mistral, DeepSeek) ship their tokenizer with the weights, so counts can be exact too.
• Anthropic doesn’t publish Claude’s tokenizer, so any offline number for Claude is an estimate. Its free count_tokens endpoint gives the official count, which Anthropic says can differ slightly from a real request.
• Google offers a free countTokens endpoint for Gemini, and its Python SDK (google-genai) can also count text locally for supported Gemini models with its LocalTokenizer.

Counts can shift even within one family: Anthropic says Claude Opus 4.7 and later use a newer tokenizer that turns the same text into about 30% more tokens than earlier Claude models. That’s why our token counter doesn’t guess for Claude and Gemini: it shows OpenAI’s count marked for reference only, and fetches the official count when you add your own key.

Why tokens matter: cost and context#

Tokens decide two things you care about:

1. Cost. APIs charge per million input tokens and per million output tokens, and output is usually several times more expensive. The LLM cost calculator turns token counts into a monthly bill.
2. Context. A model’s context window is the most tokens it can handle in one request: your prompt, the conversation so far, any documents, and the reply. The context window checker tells you whether a document fits.

In a chat or agent, the whole conversation is sent again on every turn. A 20-turn conversation therefore pays for the first message 20 times, which is why long sessions get expensive and why prompt caching exists.

How to use fewer tokens#

• Send only what the model needs: trim logs, strip boilerplate, and summarise long histories instead of resending them.
• Minify JSON you send as input; whitespace and indentation are tokens too.
• Ask for short output when that’s all you need. Output tokens cost the most.
• Use prompt caching for long, repeated prefixes like system prompts and documents.
• Pick a newer tokenizer for non-English text when you have the choice.