LLM Tokenization in Depth: BPE, Byte-Level BPE, and SentencePiece

Every benchmark score, every multilingual capability, every code-completion quality is downstream of one decision made before a single weight is trained: how you split text into tokens.

A tokenizer converts a raw string into a sequence of integer IDs. The model never sees characters. It sees IDs. Those IDs index into an embedding table, and the entire attention mechanism operates over that sequence. This means:

• A word split into 1 token costs 1 attention slot; split into 8 tokens, it costs 8 — and the model must learn to reassemble meaning across those slots.
• Token boundaries determine what the model treats as an "atomic" unit. Misaligned boundaries for non-English scripts, numbers, or code identifiers create systematic blind spots.
• Context length (e.g., 128k tokens) is measured in tokens, not characters — so a Japanese user with the same byte length as an English user gets dramatically less effective context.

Understanding the algorithm is not academic. It predicts failure modes before you run a single experiment.

Byte-pair encoding was originally a data-compression algorithm (Gage, 1994), adapted for NLP by Sennrich et al. (2016) in their seminal paper on neural machine translation.

The training procedure:

1. Start with a character-level vocabulary (every Unicode character that appears in the corpus, plus a special end-of-word marker).
2. Count every adjacent pair of symbols in the corpus.
3. Merge the most frequent pair into a new single symbol. Add it to the vocabulary.
4. Repeat steps 2–3 until the vocabulary reaches its target size.

At inference, the learned merge rules are applied in the same priority order to any input string. The result is a vocabulary of subwords — common words become single tokens, rare words decompose into smaller pieces.

Key property: BPE is a greedy, deterministic algorithm. Given the same merge rules, the same string always produces the same token sequence. No randomness at inference time (unlike some SentencePiece modes).

Heads up: the original Sennrich BPE works on Unicode characters. This causes an out-of-vocabulary (OOV) problem: if a character never appeared in training, it cannot be tokenized.

OpenAI's GPT-2 (2019) introduced a critical fix: start from bytes, not Unicode characters. Since every string is valid UTF-8, and every UTF-8 string decomposes into bytes 0–255, the base vocabulary is exactly 256 symbols — no OOV ever.

The BPE merge process then runs identically on byte sequences rather than character sequences. A Chinese character like 你 (UTF-8: 0xE4 0xBD 0xA0) starts as three byte tokens; frequently co-occurring byte sequences merge into subword tokens.

Llama 1/2 uses the same byte-level BPE via HuggingFace's tokenizers library. Llama 3 switches to a 128k vocabulary (from 32k) but keeps byte-level BPE as the foundation.

from tokenizers import Tokenizer

tok = Tokenizer.from_pretrained("gpt2")
output = tok.encode("Hello, 世界! 2+2=4")
print(output.tokens)
# ['Hello', ',', 'Ġ世', '界', '!', 'Ġ2', '+', '2', '=', '4']
print(output.ids)
# [15496, 11, 995, 30590, 0, 362, 10, 17, 28, 19]

The Ġ prefix (byte 0xC4 0xa0, displayed as Ġ) represents a leading space — GPT-2's way of encoding word boundaries without a special marker.

SentencePiece (Kudo & Richardson, 2018, Google) approaches the problem differently. Where BPE requires pre-tokenized input (whitespace-split words), SentencePiece operates directly on raw sentences — no pre-tokenization step. This matters enormously for languages like Japanese, Chinese, or Thai that have no spaces.

SentencePiece supports two algorithms:

• BPE mode — the same merge algorithm, but running on raw byte/character sequences rather than pre-split words.
• Unigram mode — starts from a large candidate vocabulary and iteratively prunes symbols whose removal increases corpus likelihood the least, keeping the vocabulary that best explains the data under a unigram language model.

Both T5 (Google) and LLaMA use SentencePiece. The T5 tokenizer uses the Unigram variant with a 32k vocabulary. LLaMA 1/2 uses SentencePiece with BPE.

import sentencepiece as spm

sp = spm.SentencePieceProcessor(model_file="llama2.model")
sp.encode("Hello world", out_type=str)
# ['▁Hello', '▁world']
sp.encode("日本語", out_type=str)
# ['▁', '日本', '語']  ← notice the asymmetric split

The ▁ (U+2581, lower one-eighth block) marks a space prefix — SentencePiece's equivalent of GPT-2's Ġ.

Vocabulary size is the single most consequential hyperparameter in tokenizer design. Here's what moves when you change it:

Larger vocabularies use fewer tokens per sentence — reducing sequence length, saving compute, and improving effective context. But they also require more parameters in the embedding and output projection layers, and rare tokens receive sparse gradient updates during training, leading to underfit embeddings.

GPT-2 used 50,257 tokens (50,000 merges + 256 bytes + 1 special token). GPT-4 uses ~100k. Llama 3 jumped from 32k to 128k specifically to improve code and multilingual performance.

Heads up: a vocabulary optimized for English will systematically under-serve other scripts. A Spanish word typically tokenizes into 1–2 tokens; an equivalent Amharic or Telugu word may take 6–12, burning through context and creating inference cost disparity.

The "fertility" of a tokenizer — average tokens per word for a given language — is a direct measure of how expensive that language is to process. Petrov et al. (2023, "Language Model Tokenizers Introduce Unfairness between Languages") quantified this rigorously.

With the GPT-4 tokenizer (cl100k_base):

• English: ~1.3 tokens/word
• Spanish: ~1.5 tokens/word
• Swahili: ~4.4 tokens/word
• Burmese: ~11.5 tokens/word
• Yoruba: ~12+ tokens/word

The consequences are not just cost. High-fertility languages:

1. Exhaust context windows faster — an identical passage in Yoruba takes 10× the token budget of English.
2. Force the model to predict more steps per semantic unit, compounding error.
3. Are systematically underrepresented in pretraining data, so their subwords get fewer gradient updates.

The fix is not trivial: a multilingual-balanced vocabulary (like mT5's 250k sentencepiece vocab) helps, but requires training data that is itself balanced — which conflicts with the reality that English dominates the web.

Code has specific tokenization pathologies that English prose doesn't:

Indentation. Python's 4-space indent can tokenize as a single token, or as four separate tokens, depending on the vocabulary. If the model sees each space separately, it must learn to count — a task attention is not built for.

Long identifiers. calculateCumulativeMovingAverage splits into many fragments. The model must reassemble the semantic meaning of the identifier from its parts. Camel case and snake case fragment differently.

Numbers. This is the worst case. The number 12345 may tokenize as 123 + 45, 1 + 2 + 3 + 4 + 5, or 12345 as one token — with no consistency. Arithmetic performed token-by-token is structurally misaligned with arithmetic performed digit-by-digit.

import tiktoken
tok = tiktoken.get_encoding("cl100k_base")  # GPT-4 tokenizer

tok.encode("12345")    # [12345]     — lucky, one token
tok.encode("123456")   # [12345, 6] — splits arbitrarily
tok.encode("1234567")  # [1234567]  — back to one token

This irregularity is why LLMs reliably fail at multi-digit arithmetic without chain-of-thought or tool use — the tokenizer breaks the digit structure that arithmetic depends on.

Beyond subwords, tokenizers reserve slots for special tokens that act as control signals. These are never produced by BPE merges — they are injected by the tokenizer wrapper and carry meaning the model learns during fine-tuning.

Common examples:

Special tokens are added to the vocabulary at fixed IDs that fall outside the BPE merge range. In HuggingFace transformers, you add them via tokenizer.add_special_tokens({"additional_special_tokens": ["<|tool_call|>"]}) and then call model.resize_token_embeddings(len(tokenizer)) to extend the embedding table.

Heads up: if you fine-tune without adding a special token to the vocabulary, the model will see it split into subwords — completely breaking its intended function as a control signal.

Different tokenizers produce wildly different segmentations for the same input. Here's a live comparison:

import tiktoken
from transformers import AutoTokenizer

text = "LLM tokenization affects everything: 日本語, def add(a,b): return a+b"

# GPT-4 (cl100k_base)
gpt4 = tiktoken.get_encoding("cl100k_base")
print("GPT-4:", gpt4.decode_tokens_bytes(gpt4.encode(text)))

# Llama 3
llama3 = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
print("Llama3:", llama3.tokenize(text))

# T5 (SentencePiece Unigram)
t5 = AutoTokenizer.from_pretrained("t5-base")
print("T5:", t5.tokenize(text))

Expect GPT-4 to handle the Japanese in 4–6 tokens, Llama 3 (128k vocab) to be similar or slightly better, and T5 (32k vocab) to fragment it more aggressively.

For production use, always measure fertility on your actual domain data before committing to a tokenizer. A 10% fertility difference across 1B tokens is 100M extra forward passes.

For domain-specific applications — biomedical text, legal documents, code-only models — training a custom tokenizer on your corpus is often worth the effort.

from tokenizers import ByteLevelBPETokenizer

tokenizer = ByteLevelBPETokenizer()
tokenizer.train(
    files=["corpus.txt"],
    vocab_size=32000,
    min_frequency=2,          # a pair must appear ≥2 times to be merged
    special_tokens=["<s>", "</s>", "<unk>", "<pad>", "<mask>"]
)
tokenizer.save_model("./my-tokenizer")

Practical tips:

• Use a corpus that mirrors your inference distribution. A general-web tokenizer trained on Common Crawl will fragment domain jargon (e.g., CRISPR-Cas9 or querySelector) aggressively.
• min_frequency=2 prunes noise; for small corpora raise it to 5–10.
• After training, run a fertility audit: compute total_tokens / total_words on a held-out validation set across all target languages/domains.
• If you're building on top of a pretrained model, do not retrain the tokenizer — the embedding table is tied to it. Instead, add targeted special tokens.