Language models read text neither letter by letter nor word by word, but in tokens. This unit plays a bigger role in everyday work with AI than it might seem at first: the billing of many AI services, the maximum length of inputs, and the length of responses are measured in tokens.
How does splitting into tokens work?
Before a large language model can process a text, what is called a tokenizer splits it into building blocks. Each building block corresponds to an entry in a fixed vocabulary and is represented internally as a number. Common words are often a single token, while rare or long words are split into several parts. A German compound word such as "Datenschutzfolgenabschätzung" (data protection impact assessment) therefore usually consists of several tokens. Spaces, punctuation marks, and digits are also captured as tokens or as part of them.
Methods such as byte pair encoding are widespread, in which character sequences that frequently occur together are merged into one token. Because many models were trained predominantly on English texts, their vocabulary is often optimized for English. German texts then often result in more tokens than comparable English ones. How many tokens a particular text produces always depends on the respective model.
The model also works in tokens when generating output: it produces its answer token by token.
Why tokens matter
- Costs: Many AI services bill by input and output tokens, often at different prices for each.
- Limits: A model's context window is specified in tokens. It determines how much text a single request may contain.
- Response length: The maximum length of a response is also limited by a number of tokens.
- Speed: The more tokens are generated, the longer a response takes.
What companies should keep in mind
- Estimate consumption: Before broad deployment, measure how many tokens typical documents and requests generate.
- Choose context deliberately: Send only the truly relevant passages instead of entire document collections. This reduces costs and often improves quality.
- Set limits: For automated applications, define upper limits for tokens per request and a budget.
- Compare models fairly: Token counts and prices are not directly comparable between models, because each tokenizer splits text differently.
How it differs
In IT security, "token" means something else: a digital proof of a sign-in or authorization, for example in single sign-on, or a small device that generates one-time codes for signing in. This has nothing to do with tokens in the sense of language models.
View AI consulting