tokenization

A rigorous tour of how modern LLMs split text into tokens — covering byte-pair encoding, GPT-2/Llama's byte-level variant, SentencePiece, vocabulary design trade-offs, and why your tokenizer silently determines multilingual fairness, code quality, and arithmetic ability.

Before a compiler can understand your code it has to chop it into meaningful pieces. Learn how scanners work, how regular expressions become finite automata, why maximal munch is the rule, and what a real token stream looks like — the foundation every later compiler phase depends on.