A token is the unit of text a large language model actually processes. Before a model sees your words, a tokenizer splits them into pieces: common words are often one token, rarer words are split into several, and punctuation and spaces count too. Everything about how a model handles text, from how much it can read at once to how much an API call costs, is counted in tokens rather than words or characters.
How tokenization works
Each model family ships with its own tokenizer, built from a fixed vocabulary of text fragments. When text comes in, the tokenizer breaks it into the fragments in that vocabulary and turns each into a number. The model works on those numbers and produces new ones, which are turned back into text on the way out.
Because vocabularies differ, the same sentence can produce a different token count in different models. English prose tends to tokenize efficiently. Code, URLs, product names, numbers and many non-English languages often need more tokens per word.
Why tokens matter for search and content
AI search systems retrieve passages and feed them to a model inside a limited context window, which is measured in tokens. Long, padded pages cost more tokens to include, and a system working to a budget may keep only part of a page. Concise passages that state the answer early are cheaper to use.
Tokens also explain some odd model behavior. A model that has never seen your brand name as a single unit sees it as several fragments, which is one reason new or invented product names are sometimes misspelled or confused with similar words in AI answers.
- Context limits: how much text fits in one request.
- Pricing: API usage is billed per input and output token.
- Output length: maximum response size is set in tokens.
- Retrieval budgets: how many passages an AI search system can pass to the model.
Example
Counting tokens for a support assistant
Quillbrook, a fictional invoicing tool, wants its support bot to read three help articles per answer. One article is a 4,000-word guide; the others are short. Running them through the vendor's tokenizer shows the long guide alone uses most of the budget. The team splits it into focused articles so the bot can include the relevant one without crowding out the rest.
Common mistakes
Assuming a token equals a word
The ratio varies by language, model and content type. Measure with the tokenizer for the model you use instead of estimating from word counts.
Ignoring token cost in page design
Boilerplate, repeated navigation text and long preambles all consume tokens if a system ingests the page as text. Keep the main content easy to isolate.
Questions
It depends on the tokenizer and the text. For ordinary English prose, a token is often a bit shorter than a word, but code, numbers and other languages can differ a lot. Use the model vendor's tokenizer tool to measure.
Not directly as a ranking signal, as far as anything published says. They affect how much of a page can be passed to the model once it is retrieved.
Sources
