Token (in language models)
- AI basics
- Standard term
A token is a unit of text an AI model reads or writes, such as a word, word part, or punctuation. The model is the trained AI inside the app. Many usage charges and text limits count tokens.
Software called a tokenizer divides text into pieces the model recognizes. A familiar word may be one piece; an unusual word or code may take several. Different models can divide the same text differently. Counting words alone therefore gives only a rough estimate of the token count.
Where you’ll meet it
You're pricing an app that drafts replies to 1,000 customer emails a day. The provider, the company running the model, charges for incoming text, called input, and text the model writes back, called output. Your estimate counts only the words in the replies.
What to do
Try a few typical emails and check the provider's input and output token counts. Input includes the emails, instructions, and earlier messages sent to the model. Apply the two rates separately so your estimate covers reading that text as well as writing the replies.
Easy to mix up
| Token (in language models) | Access token |
|---|---|
| Representing a piece of text the model reads or writes | Giving software permission to use an account or service |
Check yourself
Two models process the same product description. One charges half as much per input token but counts twice as many input tokens. Ignoring replies and other charges, which costs less for that description?
Go deeperSubword methods handle unfamiliar words, and token lengths vary across languages
Sennrich, Haddow, and Birch's 2015 paper, published at the Association for Computational Linguistics conference in 2016, adapted byte-pair encoding to represent rare words through smaller units. The method starts with small pieces and repeatedly joins neighboring pieces that often appear together. A model can then process unfamiliar words through known pieces without needing a separate token for every possible word.
Petrov and colleagues' 2023 paper, “Language Model Tokenizers Introduce Unfairness Between Languages,” found that translations of the same text can have markedly different token lengths. Comparing token prices across languages or models also requires checking how the text is divided.
Sources
- Neural Machine Translation of Rare Words with Subword Units, Sennrich et al., ACL 2016
- Language Model Tokenizers Introduce Unfairness Between Languages, Petrov et al., NeurIPS 2023
- Machine Learning Glossary, Google for Developers