Lessons

Two failure modes that survive perfect methodology: a prediction objective that pulls research toward the average consumer, and a feedback loop where synthetic data degrades the evidence base. Plus the guidelines for using this responsibly.

Silicon samples require dozens of defensible setup choices, and those choices change the answer. The study that generated 252 configurations, found correlations ranging from .23 to .84, and what it means for anyone reporting synthetic results.

The failure modes that show up when silicon samples are checked against real human data: flattened variance, wrong-signed estimates, portrayal rather than simulation, and the coverage bias that no sample size fixes.

Where the idea of using language models as synthetic survey respondents came from, the three founding results that made marketing take it seriously, and the economics that make it so tempting.

Where discrete diffusion goes after language: unified multimodal models, sequence design in biology, the fixed-length problem nobody has cleanly solved, and why alignment and evaluation both need rebuilding.

Why bidirectional attention breaks the KV cache, how block-wise approximate caching brings it back, and the conditional-independence problem that decides how many tokens you can safely unmask at once.

What it takes to train a diffusion language model to billions of parameters: the from-scratch recipe, adaptation from an autoregressive checkpoint, the supervision economics, and the capability that bidirectional training gets for free.

The formal machinery under masked text generation: categorical transition matrices, the kernel choices that were tried and discarded, the variational bound, and the two parameterizations that made the objective trainable at scale.

Turning interpretability stories into tested claims: activation patching, model editing, steering vectors, the attribution graphs that trace circuits in production models, and the problems the field has not solved.

How an unsupervised model trained on activations pulls features back out of superposition, what it found when scaled to a production language model, and where the method still breaks down.

How attention heads read and write to a shared residual stream, how they compose into circuits, and the two best-understood examples: the induction head behind in-context learning and the indirect object identification circuit in GPT-2.

Why individual neurons are the wrong unit for understanding a neural network, and what the superposition hypothesis says is really going on inside the activations.

The monotone, cardinality-constrained case is only the entrance. This lesson maps the rest: why submodular minimization is easy while maximization is hard, how to maximize non-monotone functions, what to do under matroid and knapsack constraints, and where submodularity keeps appearing across machine learning.

The theory pays off because so many real problems are submodular in disguise. This lesson works through the landmark applications: near-optimal sensor placement, influence maximization in social networks, and document summarization, each reduced to monotone submodular maximization and solved by the greedy algorithm.

Maximizing a submodular function is NP-hard, yet the simplest possible algorithm gets provably close to optimal. This lesson presents the greedy algorithm, the celebrated (1 - 1/e) guarantee of Nemhauser, Wolsey, and Fisher, why that bound cannot be beaten, and the lazy trick that makes greedy fast at scale.

Many selection problems share one property: each new item helps less than the last. Formalized, that property is submodularity, and it is what makes otherwise intractable problems solvable with guarantees. This lesson defines it precisely, shows the everyday examples, and explains why it is called the discrete analog of convexity.

The same hierarchy that classifies grammars can classify neural networks. This lesson places recurrent networks and transformers on the Chomsky ladder, explains why a transformer is theoretically weaker than it looks, and shows how formal language theory predicts where today's language models generalize and where they break.

Words have internal structure, and modelling it is computational linguistics' cleanest success. This lesson covers morphemes and the ways languages build words, why the finite-state machines at the bottom of the hierarchy are enough, and how finite-state transducers analyze and generate word forms, up to the modern neural shared tasks.

How much computational power does a language need? The Chomsky hierarchy answers it with four nested classes of grammar. This lesson builds the ladder from regular to recursively enumerable, shows the machine that recognizes each rung, and locates human language on it using the classic proof that it is not context-free.

Computational linguistics treats human language as something to be modelled precisely, and language fights back at every level. This lesson maps the levels of linguistic structure, shows why ambiguity, recursion, and Zipf's law make naive approaches fail, and frames the formal and statistical traditions the rest of the path builds on.

The theory becomes useful through libraries, and its flagship job is tuning machine-learning models. This lesson covers that application, the Tree-structured Parzen Estimator behind Hyperopt and Optuna, the tools you would actually reach for, batching, and the honest limits of the method.

A Gaussian process gives a mean and an uncertainty everywhere; the acquisition function turns them into one decision about where to spend the next expensive evaluation. This lesson covers Expected Improvement, GP-UCB and its no-regret guarantee, and Thompson sampling, and how each resolves exploration against exploitation.

Bayesian optimization needs a model that predicts the objective everywhere and knows how unsure it is. The Gaussian process delivers exactly that. This lesson builds the GP as a distribution over functions, shows how a kernel encodes assumptions, and explains why its posterior variance is the engine of the whole method.

Some functions cost hours or dollars to evaluate once, give no gradient, and must be optimized in as few tries as possible. This lesson sets up that problem, shows why grid and random search waste the budget, and introduces the surrogate-model loop that Bayesian optimization is built on.