tokenization

Search does not scan documents; it looks up precomputed answers. This lesson builds the inverted index from first principles, then covers the pipeline that feeds it: tokenization, normalisation, stemming and synonyms, and why an analysis mistake made at index time cannot be fixed at query time. Includes the classic failure where a product SKU becomes unfindable.

Before a model can process speech it has to be discretised, and audio resists that harder than text does. This lesson covers why raw waveforms are the wrong representation, how neural audio codecs learn a discrete one, what residual vector quantization actually does, and the arithmetic that governs every speech model: a minute of talking is around 195 text tokens or 36,000 audio tokens.

Cross-border business payments are slow and expensive for structural reasons: correspondent banking, trapped pre-funded capital, and messaging that is not settlement. This lesson explains why, what stablecoins actually change, how tokenized treasuries and deposits fit, and how to read the eye-watering volume statistics honestly.

How do you let software spend your money without handing it your card? This lesson covers the mechanism behind agentic checkout: scoped single-use credentials like the Shared Payment Token, Google's AP2 intent and cart mandates as signed verifiable credentials, agent identification, and the guardrails that bound what an agent may buy.

You cannot understand agentic checkout without understanding checkout. This lesson builds the payment foundation: the five parties in every card transaction, the authorization-capture-clearing-settlement path, who really pays interchange, why card-not-present fraud shapes everything, and the tokenization idea that agent payments are built on.

A rigorous tour of how modern LLMs split text into tokens — covering byte-pair encoding, GPT-2/Llama's byte-level variant, SentencePiece, vocabulary design trade-offs, and why your tokenizer silently determines multilingual fairness, code quality, and arithmetic ability.

Before a compiler can understand your code it has to chop it into meaningful pieces. Learn how scanners work, how regular expressions become finite automata, why maximal munch is the rule, and what a real token stream looks like — the foundation every later compiler phase depends on.