How Many Tokens Is a Word? Tokens Explained with Real Counts

A token is a chunk of text that a language model reads and writes, often a whole short word, sometimes part of a longer one. For English, vendors' rules of thumb put one token at about 4 characters, or 100 tokens at about 60 to 80 words, so a word averages a little more than one token.

That ratio holds for plain English prose. Other languages, numbers and code use more tokens per word, and two models can count the same text differently. The rest of this article shows real counts so you can see where the rule of thumb breaks.

What a token is

Models do not see letters or words. Before your text reaches the model, a tokenizer splits it into pieces from a fixed vocabulary and replaces each piece with an integer ID. The model predicts the next ID, and the tokenizer turns the IDs back into text.

Most current tokenizers use byte pair encoding (BPE). The tiktoken README describes BPE as a way of converting text into tokens that is reversible, works on arbitrary text, and compresses it. In practice, common words become one token, and rarer words are built from several common fragments.

OpenAI's cookbook shows the string tiktoken is great! split by the cl100k_base encoding into 6 tokens:

[b't', b'ikt', b'oken', b' is', b' great', b'!']

Two details are visible here. The unfamiliar word "tiktoken" takes three tokens, and the space before "is" and "great" is part of the token rather than a token of its own.

The rules of thumb, and who publishes them

Converted to words, Google's figure means 1 English word ≈ 1.25 to 1.67 tokens (100 ÷ 80 and 100 ÷ 60). Use it for rough budgeting only. For billing, context limits or anything you will be held to, count the actual text with the model's tokenizer or the vendor's counting endpoint.

Real counts: English, other languages and code

We counted these strings with OpenAI's tiktoken library (version 0.12.0) using the o200k_base encoding, which the OpenAI cookbook lists for GPT-4o and GPT-4o mini. You can reproduce every number with the snippet further down. Other vendors' tokenizers will give different numbers.

The English sentence is close to one token per word. The same sentence in Lithuanian takes 16 tokens for 7 words, and in Russian 20 tokens for 7 words. The Japanese sentence comes out at one token per character.

Single words and numbers show the splitting directly:

Why non-English text costs more tokens

A BPE vocabulary is built from training text. Sequences that appear often get their own tokens; rare ones are left as smaller fragments. When the text used to build the vocabulary is mostly English, English words tend to be single tokens, while words in other languages are more often split into pieces, and accented letters such as ė, ų or į can end up as separate tokens (you can see this in the Lithuanian row: lap + ė).

Newer, larger vocabularies narrow the gap. The OpenAI cookbook's own comparison shows the Japanese phrase お誕生日おめでとう taking 14 tokens in the older r50k_base encoding, 9 in cl100k_base and 8 in o200k_base. Our sentences show the same trend:

Because API prices and context windows are measured in tokens, the same content in some languages costs more and fills the context faster than in English.

Why code and data cost more tokens

Code is dense with symbols that rarely merge with neighbours: brackets, operators, semicolons, quotes. In the JSON example above, 6 "words" become 18 tokens because {", ":, ", and spaces are separate pieces, and the number 1042 splits into 104 and 2. Long digit strings, hashes, UUIDs and base64 are among the most expensive text you can send per character.

If you pass data to a model, trimming it first saves tokens. Converting a large JSON array to CSV removes the repeated key names on every row; larely's JSON to CSV converter does that in the browser.

Why token counts differ between models

Each model family has its own tokenizer, and vendors change tokenizers between generations. The same text therefore has no single "token count":

• OpenAI's cookbook shows 2 + 2 = 4 as 5 tokens in r50k_base and 7 in cl100k_base and o200k_base, which split spaces differently.
• Anthropic's token counting docs say Claude Opus 4.7 and later models use a newer tokenizer that produces approximately 30 percent more tokens for the same text than earlier Claude models, and tell you to recount prompts against the model you plan to use.
• Anthropic does not publish its tokenizer as a library, so you cannot count Claude tokens offline with tiktoken. Use Anthropic's free count_tokens endpoint, which its docs describe as an estimate that can differ slightly from the final usage.
• Google exposes count_tokens in the Gemini SDK for the same purpose.

Chat requests also carry overhead beyond your text. The cookbook's counting function adds a fixed number of tokens per message for role and formatting markers, and notes that tool definitions consume extra tokens on top.

How to count tokens yourself

For OpenAI encodings, install tiktoken and count locally:

pip install tiktoken
python -c "import tiktoken; e = tiktoken.get_encoding('o200k_base'); print(len(e.encode('The quick brown fox jumps over the lazy dog.')))"

Example output:

10

For Claude, call the counting endpoint with the same body you would send to the Messages API. Anthropic's docs show this request returning { "input_tokens": 14 } for a short system prompt and one message:

curl https://api.anthropic.com/v1/messages/count_tokens \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "content-type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{"model": "claude-opus-5-5", "system": "You are a scientist",
       "messages": [{"role": "user", "content": "Hello, Claude"}]}'

Token counts matter most when you are close to a model's limit. What a context window is explains how input and output share that limit, and how to reduce token usage covers the practical ways to send fewer of them.

Images, audio and video are tokens too

Multimodal input is converted to tokens as well, by fixed rules rather than by a text tokenizer. Google's Gemini docs, for example, say an image of 384 pixels or less in both dimensions costs 258 tokens, larger images are tiled into 768 × 768 pieces at 258 tokens each, and audio costs 32 tokens per second. Each vendor publishes its own rates, so check the docs for the model you use.

Frequently asked questions

How many tokens is 1,000 words?

For ordinary English prose, roughly 1,250 to 1,670 tokens. Google's rule of thumb for Gemini is that 100 tokens equal about 60 to 80 English words, so 1,000 words lands in that range. Other languages and source code usually need more tokens for the same number of words, so count with the model's own tokenizer when the number matters.

Is a token the same as a character?

No. A token is usually several characters long in English: Google says a Gemini token is about 4 characters, and OpenAI's tiktoken README says a token averages about 4 bytes. In scripts such as Japanese or Chinese a token can be a single character, and a single emoji or rare character can take more than one token.

Do spaces and punctuation count as tokens?

Yes, but often not as separate ones. Most tokenizers attach the space to the start of the following word, so ' quick' is one token. Punctuation marks, brackets and runs of whitespace in code are frequently tokens of their own.

Why does the API report more tokens than my text has?

Chat APIs wrap each message in formatting tokens that mark roles and turns, and they may add a system prompt or tool definitions. OpenAI's cookbook shows a fixed overhead per message, and Anthropic's docs note that a count can include tokens added for system optimizations, which are not billed.

Sources

• Google AI for Developers: Understand and count tokens (Gemini API)
• OpenAI tiktoken (BPE tokenizer) on GitHub
• OpenAI Cookbook: How to count tokens with tiktoken
• Claude API docs: Token counting