AI lessons & courses
214 lessons · 56 learning paths · free, quiz-checked, no signup required
How modern AI systems actually work, from the transformer architecture and attention up through retrieval, agents, evaluation, and deployment. The lessons are mechanism-first: what the math does, what the engineering trade-offs are, and where the published evidence stands. Written for engineers and technical leaders who want depth beyond the headlines.
Learning paths
Deep Learning on Limit Order Book Data
At a one-event horizon, 92 percent of mid-price labels are exactly no-change, so a model that always predicts flat scores 92 percent accuracy. That single number explains most of the gap between published order book results and what happens on a trading desk. This path covers what the book actually contains, what the architectures assume, why results fail to replicate, and the cost hurdle that decides whether a real edge is worth anything.
Reinforcement Learning for Trade Execution and Market Making
A classical execution schedule is an open-loop policy: it commits to a plan before seeing anything, and reinforcement learning's entire value here is closing that loop. This path measures what closing it is actually worth, shows how a mis-sized penalty makes leaving part of the order unexecuted rationally optimal, and demonstrates that one unverifiable queue assumption moves the simulated fill rate by a factor of two. Every figure is computed.
Financial Machine Learning: Building a Dataset That Does Not Lie
Shuffled cross-validation reports 54 percent accuracy on market data containing no signal whatsoever, and that number is why competent machine learning so often fails on markets. This path fixes the dataset rather than the model: labels that describe trades someone would actually place, features that are stationary without being amnesiac, and a validation scheme that returns chance when there is nothing there. Every figure in it is computed and reproducible.
AWS Bedrock and AgentCore: Agents as Managed Infrastructure
Running one agent is a while-loop around a model call. Running hundreds, for real users, with real credentials and real consequences, is infrastructure. This path covers what AWS provides at each layer: Bedrock's one door to many models, the runtime that isolates every session, the gateway that turns your existing APIs into agent tools, and the memory, identity and tracing that decide whether an agent survives production.
Recommender Systems: The Two-Stage Machine
No model ranks the whole catalogue: a cheap retrieval stage cuts millions of items to hundreds, an expensive ranker orders those hundreds, and the trap nobody warns you about is that the model trains on clicks it caused. This path builds the architecture, the two-tower geometry of taste, the objective functions that encode what a product values, and the feedback loops and exploration budgets that decide what the system becomes.
MLOps: Keeping a Model Working After You Ship It
Training a model is the part that works. The system around it is what decays, and it decays quietly: no exception, no alert, just answers that are slowly less right. This path covers what actually breaks. Why changing one feature moves every weight, the three levels of automation and which one you need, the skew between training and serving that no isolated test can see, and the monitoring that decides when a model has stopped earning its place.
FiftyOne: Curating Computer Vision Datasets You Can Actually See
Most vision projects are limited by their dataset rather than their architecture, and most teams cannot see their dataset. A hundred thousand images in a folder is not something you can inspect, so problems that would be obvious in ten minutes of looking go unfound for months. This path covers the tool built for that: the data model that makes a dataset queryable, views as chained questions, the analysis methods that rank where to look, and evaluation that turns a metric back into the images behind it.
How Voice Models Work: Tokens, Recognition, and Synthesis
One minute of talking is about 195 tokens as text and about 36,000 as audio. That ratio explains almost every design decision in speech systems, and this path starts there. How neural codecs turn a waveform into something a transformer can model, why speech recognition has three architectures rather than one and which of them can stream, how treating audio as tokens made cloning a voice from three seconds possible, and what the standard speech-to-text-to-speech pipeline throws away that it can never get back.
The Illusion of Thinking, and the Rebuttal That Followed
Apple researchers ran a careful experiment on whether reasoning models reason, found accuracy collapsing to zero past a complexity threshold, and the result travelled fast. Then a response appeared that disputed none of the data and argued the experiment measured something else: models were being asked to print answers longer than their output limits, some puzzles had no solution at all, and asking for a program instead of a move list restored accuracy on the same instances. This short path follows both, and ends with the rules for measuring anything.
A Definition of AGI: What Happens When You Actually Try to Measure It
AGI anchors company charters, national policy and enormous investment, and until recently nobody had an operational definition of it. A 2025 paper with more than thirty authors proposes one, by borrowing the framework psychology spent a century validating and scoring machines against all ten of its cognitive domains. The totals make headlines. The profile is more interesting: current systems are at full marks on some domains and at exactly zero on the ability to form a new long-term memory.
The Era of Experience: Reading Silver and Sutton's Argument
The man who wrote the textbook on reinforcement learning and the man who led AlphaGo published a short paper arguing the field has taken a wrong turn. Training on human data, they say, makes human performance the target by construction, and the next capability jump requires agents that learn from what actually happens instead. This short path takes the argument seriously: the three eras, the four things an experiential agent would do differently, and the published critique that its answer to safety does not yet work.
FlashAttention and the Art of Finding the Real Bottleneck
For five years attention was optimised by cutting FLOPs, and the faster algorithms kept not being faster. They were solving the wrong problem: attention spends its time moving a matrix between GPU memory tiers, not computing. This path follows the diagnosis and the fix, from the memory hierarchy through the rescaling identity that lets softmax be tiled, to writing fused kernels yourself, and ends where the same signature shows up next: normalisation, the optimizer step, and the logits tensor nobody thinks about.
Beyond Attention: Linear Attention, State Space Models, and Hybrids
At a million tokens a transformer's KV cache is 344 GB. A recurrent state doing the same job is 16.8 MB, and constant at any length. This path explains how that is possible and what it costs. Delete the softmax and attention becomes a recurrent network you can still train in parallel. Start instead from control theory and you arrive at the same object, which is the result Mamba-2 made precise. Then the honest part: a fixed state has fixed capacity, exact retrieval is what breaks first, and a few attention layers buy most of it back.
Synthetic Training Data: Generating It Without Poisoning the Model
Generating 50,000 training examples costs roughly a thousandth of annotating them, and cheapness is the least interesting thing about it. What decides whether the result works is a set of properties nobody notices until the model is trained: whether the generator holds the information at all, whether a checker can lift the ceiling above it, whether the dataset covers its input space or repeats one example, and whether the real data ever leaves the mix. This path builds the pipeline that survives all four.
Constrained Decoding: Output That Always Parses
A 5 percent JSON failure rate does not double your average latency, it doubles your p95. Constrained decoding removes the failure entirely by editing the logits before sampling, so invalid tokens have probability exactly zero. This path builds the mechanism from the decoding loop outward: how a schema compiles into an automaton and then into a per-token mask, why recursive grammars need a stack and how that cost is made near-free, and the part most teams get wrong, that a badly ordered schema constrains the reasoning as well as the output.
Distributed Training at Scale: FSDP, Tensor and Pipeline Parallelism
Mixed-precision Adam costs 16 bytes per parameter, so a 70B model needs 1120 GB of state before a single activation is stored. Getting that onto real hardware, and keeping the run alive while a component fails every three hours, is the skill behind every frontier model. This path builds it: where the memory actually goes, how ZeRO and FSDP shard what data parallelism duplicates, how tensor and pipeline parallelism split the computation itself, and what makes a fifty-day run finish.
Quantizing LLMs: From 16 Bits to 4 Without Breaking the Model
A 70B model is 140 GB at 16 bits and fits on no single accelerator. At 4 bits it is 36 GB, runs about four times faster per token, and leaves most of the card free for KV cache. Getting there without wrecking the model is a specific, teachable skill. This path builds it from the arithmetic up: what quantization does to a number, why large transformers grow activation outliers that break naive methods, how GPTQ and AWQ use calibration data, and how to measure the damage honestly before you ship.
LLMs in Trading: What the Evidence Actually Supports
Language models are poor at the job they are advertised for and genuinely good at one nobody mentions: turning unstructured text into structured data at a scale that was never affordable. This path locates them against the trading stack, then confronts the defect that makes almost every published result hard to read, since a model trained on the backtest period may already know what happened and uses that knowledge even when told not to. It reads the leading study qualifier by qualifier, and ends with the engineering and governance that survive an audit.
Causal Inference: From Correlation to Consequence
Every model whose output drives a decision is answering a causal question, whatever it was trained on. This path builds the machinery that makes those questions answerable: structural causal models and the three ways variables become associated, the identification results that say exactly when observational data suffices and by what formula, the hard limit on learning a graph from data, and where invariance across environments turns causal structure into a training signal.
Federated and Decentralized Learning
Hospitals, phones and banks hold the data that would train the best models, and none of them are allowed to share it. This path builds the field that trains on data you never see: the FedAvg round structure and why local steps are the communication lever, the client drift that heterogeneous data causes and the control variates that correct it, gossip protocols that remove the server entirely and the spectral gap that governs them, and what model updates actually leak once someone attacks them.
Easy, Hard, Impossible: Phase Transitions in Learning
When your model fails, are you short of data or short of computation? Those look identical from the outside and have completely different fixes. This path builds the statistical physics of computation, the field that made the distinction precise: planted models and the high-dimensional limit, the three-phase structure where information can be present while every efficient algorithm fails, the message-passing algorithms that reach the computational limit and predict their own error exactly, and what genuinely transfers to deep learning.
Mechanistic Interpretability: Reverse-Engineering Neural Networks
Attribution methods tell you which inputs mattered. Mechanistic interpretability asks the harder question: what algorithm is the network actually running? This path builds the field from its foundations, why individual neurons are the wrong unit and what superposition says is really happening, through the circuits that implement in-context learning, the sparse autoencoders that pulled 34 million features out of a production model, and the causal interventions that separate a tested claim from a plausible story.
Diffusion Language Models in Depth
A technical deep dive for anyone who already knows that diffusion language models generate text by iterative unmasking and wants to know how they are actually built. This path covers the formal machinery: categorical transition matrices and the variational bound that reduces to weighted masked language modelling, the from-scratch and checkpoint-adaptation training routes that carried the paradigm to 8 billion parameters, the approximate caching and confidence-aware decoding that finally made serving competitive, and the multimodal, safety, and evaluation problems that remain open.
Submodular Optimization: Diminishing Returns with Guarantees
Choosing the best subset under a budget is usually intractable, unless the objective has diminishing returns. That property is submodularity, and it turns hard selection problems into ones a simple greedy algorithm solves near-optimally. This path builds it from the definition and its convexity analogy, through the celebrated (1 - 1/e) guarantee of Nemhauser, Wolsey, and Fisher, to sensor placement, influence maximization, and summarization, and finally the wider landscape of minimization and non-monotone problems.
Computational Linguistics: The Formal Science of Language
How much computational power does human language actually require, and what can a neural network trained on it really compute? This path answers both with one tool. It builds the levels of linguistic structure and why language resists naive modelling, the Chomsky hierarchy that classifies grammar power, the finite-state methods that make morphology a solved problem, and the modern results placing RNNs and transformers on that same ladder. Grounded in the landmark results of Chomsky, Shieber, and the formal theory of language models.
Bayesian Optimization: Tuning Expensive Black-Box Functions
Some functions cost hours or dollars to evaluate once and give no gradient, so you must find their optimum in as few tries as possible. This path builds the standard method for that: why grid and random search waste the budget, how a Gaussian process predicts the objective and its uncertainty everywhere, how acquisition functions like Expected Improvement and the no-regret GP-UCB of Srinivas, Krause, Kakade, and Seeger decide where to look next, and how the tools work in practice. Grounded in the landmark papers of the field.
Fine-Tuning versus RAG: A Decision Framework
They are usually presented as alternatives when they change different things: fine-tuning adjusts behaviour, retrieval supplies knowledge. This cursus makes the choice properly. Why fine-tuning is a poor way to add facts, what it is genuinely good at, and the three questions that settle the matter on their own. Then what building each involves, from LoRA and the training-data problem that stalls most projects to the retrieval pipeline. Then the options the framing hides, long context, prompt caching, agentic retrieval, and how to revisit a decision that has expired.
Vector Databases in Depth
Exact nearest neighbour search is intractable at scale and k-d trees collapse above a couple of dozen dimensions, so every production system trades recall for speed. This cursus covers that trade properly. Why the classical indexes fail and how to measure recall against brute-force ground truth. Then HNSW as a navigable small-world graph, IVF as partition-and-probe, and the quantization schemes that decide your infrastructure bill. Then hybrid search, a tuning method that starts from a stated target, capacity planning, and the operational failures no benchmark shows.
Bias Detection and Fairness Testing
Fairness has several formal definitions that sound equally reasonable, and Chouldechova and Kleinberg proved you cannot satisfy them all at once. This cursus is the hands-on version. Separating the three meanings of bias, tracing the six points where disparity enters, and why removing a protected attribute hides the problem rather than fixing it. Then the metrics, the impossibility result, and choosing a criterion by asking which error harms the person more. Then running an audit that produces a documented, accepted trade-off rather than a claim to have removed bias.
Hallucinations: Detection, Grounding, and Abstention
A model trained to predict the next token has nothing in its objective that represents truth, so a fabricated citation is a plausible continuation rather than a malfunction. This cursus covers what follows: the taxonomy that tells you whether a failure is a generation or a corpus problem, why fluency carries no signal and why self-review does not work, grounding and the detection methods that compare output against something external, and abstention, including calibration, the coverage-accuracy trade, and building a system that can say it does not know.
LLM Guardrails and Red-Teaming
A language model receives instructions and data through one channel and cannot reliably tell them apart, which is why prompt injection has no equivalent of the parameterised query. This cursus builds the defence that follows from that fact: the threat model, from the OWASP Top 10 through indirect injection and the lethal trifecta; the control layer, with its five rail types, the false-positive budget that kills deployments, and why architectural containment beats filtering; and red-teaming that produces findings rather than reassurance.
Explainable AI: Making Model Decisions Accountable
A model that predicts well can still be impossible to justify, and "the algorithm decided" is not an answer to a rejected applicant, a clinician, or an auditor. This cursus builds the field from the mechanism up: the taxonomy and the global tools (permutation importance, partial dependence, ICE), then local attribution with LIME and SHAP including the Shapley axioms that make SHAP unique, then Integrated Gradients, Grad-CAM, and counterfactual explanations. It ends with the harder question: explanations can fail silently or be deliberately faked, so how do you check yours?
Energy-Based Models: One Lens for Machine Learning
Energy-based models replace input-to-output functions with a scalar energy that scores how compatible a configuration is, the framework Yann LeCun has advocated for decades. This cursus builds it: the energy landscape and inference as finding the lowest-energy answer, how to train one when the partition function is intractable (contrastive divergence, noise-contrastive estimation, score matching, and regularized methods), and the unifying view in which classification, self-supervised learning, JEPA, and diffusion all become one idea.
Self-Supervised Learning: How Machines Learn Without Labels
Labels are the bottleneck of deep learning, and self-supervised learning is how models learn from unlabeled data instead, the approach Yann LeCun calls the dark matter of intelligence. This cursus builds it from the ground up: pretext tasks and the shift to joint embeddings, the collapse problem that lurks underneath, contrastive methods that fix it with negatives (SimCLR, MoCo), and the negative-free families that fix it without them (BYOL, SimSiam, VICReg, Barlow Twins, MAE), with the through-line to JEPA and the energy-based view.
JEPA: Learning by Predicting Representations
Most self-supervised models either reconstruct pixels, wasting capacity on detail nobody can predict, or contrast augmented views, baking in hand-crafted bias. JEPA takes a third path: predict the representation of what is hidden, so the model can discard the unpredictable. This cursus builds it from the ground up: why predicting representations beats predicting pixels, then the machine itself, encoders, predictors, and the collapse problem that asymmetry and stop-gradients defeat, and finally I-JEPA and V-JEPA 2, where the idea becomes a world model that plans a robot's actions.
AI for Students: Study Smarter Without Cheating
AI can be the best study partner you have ever had, or a shortcut that quietly stops you from learning. This cursus shows you the difference. Learn the mindset that separates using AI to learn from using it to outsource thinking, the concrete study techniques that turn AI into a personal tutor, explaining, quizzing, and feedback, and how to use AI honestly so that the honest path and the path that actually makes you smarter turn out to be the same one.
AI for Teachers: Save Time and Teach AI Literacy
A practical guide for educators to reclaim time with AI without losing what makes teaching human. Learn the categories of teaching AI tools and their caveats, how to generate lesson plans, materials, differentiation, and feedback with the right prompts and review, and how to teach in a world where students have AI too: why detectors fail, how to design meaningful assessment, and how to teach AI literacy while protecting privacy and equity.
Voice AI Agents: How They Actually Work
Voice AI agents you can call and talk to like a person are quietly reshaping customer service and phone support, and they are more clever engineering than magic. This cursus explains how they really work: the streaming speech-to-text, LLM, and text-to-speech pipeline and why latency is everything, the turn-taking and interruption handling that makes a conversation feel human, and the architecture, tools, failure modes, and evaluation behind a production voice agent. Clear, current, and mechanism-first.
The trillion-dollar AI buildout
AI stopped being only a software story and became one of the largest construction efforts in history: warehouses of specialized chips, new power plants, and hundreds of billions in yearly spending. This cursus explains it from the ground up, why compute became the bottleneck, what an AI data center really is, why electricity is now the binding constraint, and how to judge whether the whole thing is a rational bet or a bubble. Mechanisms, not predictions.
How AI is actually used in social media
AI does three unrelated jobs on a social platform, and mixing them up causes most of the confusion. Learn the map first: what decides your feed, what generates the content, and what takes it down. Then get an honest account of where generative tools genuinely help and where they invent facts with total confidence, and why nobody, including the detectors sold for the job, can reliably tell you whether something was made by AI. No hype, no tactics, just what the machines do and how they fail.
How social media algorithms decide what you see
The same machine runs behind YouTube, TikTok, Instagram and Pinterest, and once you see it the feed stops being mysterious. Learn the two-stage funnel that picks ten items out of a hundred million in a tenth of a second, why optimizing for clicks produces clickbait as a correct answer, and how content embeddings let a post from a zero-follower account reach millions. Finish able to explain any feed's behavior from its objective, and to tell real mechanisms from the folklore creators repeat.
Generative AI: diffusion and flow models
How modern image, video, and audio generators actually work, from first principles to the frontier. Learn the denoising mechanism behind diffusion models, the guidance and sampler tricks that turn them into products, the flow-matching reframing that powers state-of-the-art generators, and how diffusion is being rebuilt for text. Finish able to read any generative-model release mechanically: corruption, path, target, and where the inference compute goes.
Loop engineering for LLM agents
Build agent loops that finish, recover from errors, stay on budget, and produce checkable work. Start with the smallest ReAct loop; layer on state management, error taxonomy, and budgets; finish with adversarial verification panels, sub-agent orchestration, and the loop-shaped anti-patterns that ship to prod more than they should.
LLM, agent, and RAG evaluation & monitoring
A compact track on how to know your LLM-powered system is actually working: LLM benchmarks vs. real evals, agent trajectory evaluation, RAG evaluation in production, and the OpenTelemetry GenAI conventions that tie traces across LangSmith, Phoenix, and Datadog LLM.
Reinforcement Learning today
A short, modern tour of where RL is actually working in 2026: the post-RLHF landscape, DPO and its variants, RL with verifiable rewards (GRPO, DeepSeek-R1 style), and how reasoning models use RL on chains of thought.
How LLMs work: from scratch
Ten lessons that build up a modern LLM end to end — neural nets and backprop, attention and transformers, tokenization, pretraining, modern architectures and MoE, scaling laws, post-training (SFT/RLHF/DPO), and inference internals like the KV cache.
Harnesses, MCP, and advanced agent techniques
How modern coding agents are actually built: the Model Context Protocol, MCP server authoring in Python and TypeScript, agent harness design, subagent orchestration, tool-use patterns, and context engineering for long-running agents.
Advanced RAG: graphs, hybrid retrieval, and evaluation
Nine lessons that take Retrieval-Augmented Generation past the cosine-search baseline. Chunking, hybrid retrieval with rerankers, query rewriting, GraphRAG and HippoRAG/RAPTOR, agentic multi-hop patterns, and how to actually evaluate the whole thing with RAGAS.
Reinforcement Learning Foundations
Go from zero to deep RL in four lessons. You will formalize sequential decision-making as a Markov Decision Process, solve small MDPs exactly with dynamic programming, implement Q-learning and SARSA from scratch, and understand DQN, policy gradients, actor-critic, and PPO well enough to use them in real projects.
Deep Learning Foundations
Build and train neural networks from scratch. By the end you will implement forward and backward passes in NumPy, tune optimizers and regularizers to close the train-val gap, design convolutional architectures for image tasks, and read transformer papers fluently — understanding self-attention, multi-head attention, and positional encodings from first principles.
AI 3D Mesh Generation: Foundations to Production
A five-lesson tour of how Meshy, Tripo, Rodin, Hunyuan3D, and Trellis actually work in 2026. Starts from the 3D representation zoo, walks through the modern image-and-text-to-3D pipeline, dives into native 3D generative models, covers texturing and PBR, and ends with the production pipeline and platform landscape.
LLM Models in Practice
Six lessons on choosing and using the right LLM for the job. Get a clear-eyed view of the 2026 model landscape, learn to compare capabilities honestly, understand the open-vs-closed tradeoff, and stop optimizing for benchmarks that don't predict your task.
LLM Basics
A single-lesson taster that introduces the most fundamental concept behind every modern LLM app: the context window. Learn what tokens are, why models forget, and what 'lost in the middle' actually means in practice.
Modern AI Beyond Text
Five lessons on AI that sees, hears, and generates. Start with the unifying idea of multimodal models, dive into vision-language models and OpenCLIP, watch video generation make the leap from research to product, then bring it home with AI-aided software development.
How Agentic AI systems work
Move beyond single prompts into agents that plan, use tools, and loop. Six lessons covering the agent mindset, the LangGraph framework, the realities of context windows, where humans belong in the loop, the security pitfalls, and how to evaluate non-deterministic systems.
How LLM apps work
A practical seven-lesson path from raw model calls to a production-ready RAG application. You'll learn how to ground LLMs in your own data, orchestrate the pieces with LangChain, observe what they actually do, and evaluate the result so you can ship with confidence.
All AI lessons
Why Correlation Is Not Enough
The formal machinery that makes causal questions answerable: structural causal models, graphs as assumptions you can inspect, the three ways variables become associated, and why prediction and intervention are different problems.
Privacy Attacks, Real Guarantees, and Open Models
What model updates actually leak, the two mechanisms that turn data minimisation into a quantified guarantee, poisoning by malicious clients, and what the Apertus project shows about openness at the other end of the spectrum.
Removing the Server: Gossip and Decentralized SGD
What happens when nobody coordinates: averaging by talking only to neighbours, why the graph's spectral gap sets the convergence rate, and how compressing messages by orders of magnitude still converges.
Client Drift: The Heterogeneity Problem
Why local training on non-identical data pulls clients apart, how averaging their updates produces a model that suits nobody, and the control-variate fix that corrects the drift.
Training on Data You Are Not Allowed to See
The federated setting: why hospitals, phones, and banks cannot pool their data, what changes when the training loop crosses a network, and the FedAvg algorithm that made the idea practical.
What Phase Transitions Mean for Machine Learning
Taking the framework beyond solvable toy models: sharp transitions in real learning, why the loss landscape of a neural network is not the fractured one theory warns about, and what the physics lens genuinely explains.
Message Passing and the Algorithms That Reach the Limit
Belief propagation, the cavity method, and approximate message passing: how physics-derived algorithms achieve the best performance any efficient method can, and how state evolution predicts their behaviour exactly before you run them.
Easy, Hard, and Impossible: The Three Phases
The central result of the field: problems split into three regimes as data increases, and the middle one contains enough information to solve them while no efficient algorithm can. Community detection and planted clique make it concrete.
Planted Problems and the High-Dimensional Limit
Why statistical physics has anything to say about algorithms: planted models with known ground truth, the large-system limit where randomness stops fluctuating, and the Bayes-optimal benchmark that makes hardness measurable.
Beyond Text: Multimodal, Safety, and Open Problems
Where discrete diffusion goes after language: unified multimodal models, sequence design in biology, the fixed-length problem nobody has cleanly solved, and why alignment and evaluation both need rebuilding.
Making Diffusion LLMs Actually Fast
Why bidirectional attention breaks the KV cache, how block-wise approximate caching brings it back, and the conditional-independence problem that decides how many tokens you can safely unmask at once.
Training Diffusion Language Models at Scale
What it takes to train a diffusion language model to billions of parameters: the from-scratch recipe, adaptation from an autoregressive checkpoint, the supervision economics, and the capability that bidirectional training gets for free.
The Mathematics of Discrete Diffusion
The formal machinery under masked text generation: categorical transition matrices, the kernel choices that were tried and discarded, the variational bound, and the two parameterizations that made the objective trainable at scale.
Causal Interventions, Steering, and What Remains Open
Turning interpretability stories into tested claims: activation patching, model editing, steering vectors, the attribution graphs that trace circuits in production models, and the problems the field has not solved.
Sparse Autoencoders and Dictionary Learning
How an unsupervised model trained on activations pulls features back out of superposition, what it found when scaled to a production language model, and where the method still breaks down.
Circuits: Reverse-Engineering Transformer Algorithms
How attention heads read and write to a shared residual stream, how they compose into circuits, and the two best-understood examples: the induction head behind in-context learning and the indirect object identification circuit in GPT-2.
Features, Directions, and Superposition
Why individual neurons are the wrong unit for understanding a neural network, and what the superposition hypothesis says is really going on inside the activations.
Beyond Greedy: Minimization, Non-Monotone, and Richer Constraints
The monotone, cardinality-constrained case is only the entrance. This lesson maps the rest: why submodular minimization is easy while maximization is hard, how to maximize non-monotone functions, what to do under matroid and knapsack constraints, and where submodularity keeps appearing across machine learning.
Submodularity in the Wild: Sensors, Influence, and Summaries
The theory pays off because so many real problems are submodular in disguise. This lesson works through the landmark applications: near-optimal sensor placement, influence maximization in social networks, and document summarization, each reduced to monotone submodular maximization and solved by the greedy algorithm.
The Greedy Algorithm and Its 63% Guarantee
Maximizing a submodular function is NP-hard, yet the simplest possible algorithm gets provably close to optimal. This lesson presents the greedy algorithm, the celebrated (1 - 1/e) guarantee of Nemhauser, Wolsey, and Fisher, why that bound cannot be beaten, and the lazy trick that makes greedy fast at scale.
Submodularity: The Mathematics of Diminishing Returns
Many selection problems share one property: each new item helps less than the last. Formalized, that property is submodularity, and it is what makes otherwise intractable problems solvable with guarantees. This lesson defines it precisely, shows the everyday examples, and explains why it is called the discrete analog of convexity.
What Neural Language Models Can Actually Compute
The same hierarchy that classifies grammars can classify neural networks. This lesson places recurrent networks and transformers on the Chomsky ladder, explains why a transformer is theoretically weaker than it looks, and shows how formal language theory predicts where today's language models generalize and where they break.
Morphology and Finite-State Methods
Words have internal structure, and modelling it is computational linguistics' cleanest success. This lesson covers morphemes and the ways languages build words, why the finite-state machines at the bottom of the hierarchy are enough, and how finite-state transducers analyze and generate word forms, up to the modern neural shared tasks.
Formal Grammars and the Chomsky Hierarchy
How much computational power does a language need? The Chomsky hierarchy answers it with four nested classes of grammar. This lesson builds the ladder from regular to recursively enumerable, shows the machine that recognizes each rung, and locates human language on it using the classic proof that it is not context-free.