72. Deep with Tokens

The lesson

Before a large language model can read or write text, the text must be converted into numbers. A tokenizer is the component that splits raw text into tokens, which are the model's basic units of meaning. Tokens are not words: they can be whole words, word fragments, punctuation marks, numbers, or even single bytes.

Different tokenizers produce different token counts for the same text. For example, the English sentence "The quick brown fox" might be five tokens in one tokenizer and seven in another. Code tends to use more tokens than natural language because brackets, indentation, and symbols are often split into separate tokens.

The most common tokenization method is Byte Pair Encoding, or BPE. BPE starts with individual characters and repeatedly merges the most frequent adjacent pairs until it reaches a target vocabulary size, often 32,000 to 200,000 tokens. GPT-style models and Claude use BPE variants. BERT and many encoder models use WordPiece, which is similar but uses a different scoring rule. T5 and Llama models often use SentencePiece or Unigram tokenization, which can handle many languages more gracefully.

A model's vocabulary includes not only content tokens but also special tokens. Examples include end-of-text markers, beginning-of-sequence markers, padding tokens, and role markers such as `<|system|>`, `<|user|>`, and `<|assistant|>`. These special tokens tell the model where a conversation turns or where a document ends.

The context window is the maximum number of tokens a model can process in one forward pass. If your input is 3,000 tokens and the model's context window is 4,000 tokens, you have only 1,000 tokens left for the response. Exceeding the limit causes truncation or an error, depending on the API. Longer context windows, such as 128,000 or 1,000,000 tokens, are becoming common but still cost more and may reduce attention quality at the extremes.

Token usage directly affects cost. Most API providers charge per input token and per output token. A verbose prompt or a long-winded response can multiply your bill. Code and structured data such as JSON or XML often consume more tokens than plain prose because of brackets, indentation, and repeated keys.

To estimate tokens, you can use online tokenizer visualizers, the provider's own tokenizer endpoint, or libraries such as tiktoken for OpenAI models. A rough rule of thumb for English is about 0.75 words per token, but this varies by language and writing style. Multilingual text and code can be much less efficient.

Tokens also affect quality. Models pay attention to all tokens, but attention is not uniform. Important instructions placed at the very beginning or end of a long prompt are often remembered better than instructions buried in the middle, a phenomenon sometimes called "lost in the middle." Therefore, put critical constraints and the most relevant context where they are most visible.

You can optimize token usage by removing redundant text, using bullet lists instead of long prose, avoiding duplicated examples, and choosing models with appropriate context windows. However, do not over-optimize at the expense of clarity: a slightly longer but unambiguous prompt usually beats a terse prompt that the model misinterprets.

Check yourself