reinforcement-learning

A market maker quotes both sides and profits from the spread, but every fill leaves an unwanted position and the counterparties who trade most eagerly are the ones who know something. This lesson simulates the inventory-skew trade-off, showing a 63 percent cut in exposure for 2.5 percent of profit, and locates where a learned policy genuinely helps.

Reinforcement learning needs an environment that responds to the agent, and historical market data is a fixed recording that does not. This lesson shows that replaying the same day gives a fill rate anywhere from 38 to 73 percent depending on an assumption the data cannot settle, and covers what to do about it.

An agent optimises the reward you wrote, not the objective you meant, and in execution the gap between those is unusually easy to open. This lesson computes how a mis-sized penalty makes leaving part of the order unexecuted rationally optimal, and covers the shaping that provably does not change the policy.

A static execution schedule is an open-loop policy: it commits to a plan before seeing anything. This lesson formulates execution as an MDP so the plan can react, and computes the ceiling on what any adaptive policy could win, which turns out to collapse as market impact grows.

The paper's answer to safety is that reward signals adapt to human feedback: a network inside the reward function learns which grounded signals to weight from how people respond. This lesson works through that proposal and the published critique arguing it fails twice over, on specification gaming and on goal misgeneralization, and what an honest reading of the disagreement leaves you with.

The paper is concrete about what an experiential agent would differ on, and names four: it lives in a continuous stream rather than episodes, acts in the world rather than emitting text, takes rewards from grounded signals rather than human judgement, and plans in terms it worked out rather than imitating human chain of thought. This lesson works through each.

David Silver and Richard Sutton argue that the current approach has a ceiling built into it, because a system trained to predict what humans wrote is aiming at human performance by construction. This lesson works through their three eras, the claim about data exhaustion, why they think superhuman performance needs a different learning signal, and the honest counter-arguments.

Why prediction systems fail when deployed, how invariance across environments becomes a training signal, and where causal reasoning enters bandits, reinforcement learning, and language models.

Where the pure feedforward story breaks: sensory entrainment, dynamic movement primitives for non-rhythmic motion, how CPGs combine with reinforcement learning in current legged robots, and the exoskeleton applications.

The simulation result that started an entire literature: independent Q-learning agents converging on supra-competitive prices with punishment strategies, nobody having programmed them to, and what the mechanism actually is.

From revenue management to reinforcement learning: the mechanics of software that sets prices, the distinction between following rules and learning a policy, and why that distinction turns out to be the one that matters.

A deep look at how reinforcement learning on chains-of-thought powers o1, DeepSeek-R1, Claude reasoning, and Gemini Thinking — covering GRPO, MCTS-style search, test-time compute scaling, and distillation into smaller models.

How Group Relative Policy Optimization turns unit tests and theorem provers into training signals — the technique behind DeepSeek-R1's leap in math and code reasoning without a reward model.

A deep dive into DPO (Rafailov et al. 2023) and its successors — how they reformulate RLHF as a classification problem, the math behind the implicit reward, and where each variant wins or loses against PPO-based pipelines.

Trace the full arc of Reinforcement Learning from Human Feedback — preference data, Bradley-Terry reward models, PPO with KL penalty, reward hacking, and why most labs have moved to offline alternatives like DPO and RAFT.

An honest map of RL in 2026 — the domains where it actually reaches production (LLM post-training, robotics policies, ad bidding, RLHF, reasoning models) and the places where it still cannot reliably cross the lab-to-deployment gap.

Tabular methods break down when the state space is continuous or astronomical in size. Learn how neural networks extend RL via DQN, how the policy gradient theorem makes it possible to differentiate through stochastic policies, and where actor-critic, PPO, and the deadly triad fit into the picture.

Leave the model behind. Monte Carlo methods wait for a full episode to update; TD methods bootstrap from the very next step. See exactly where SARSA and Q-learning diverge on the on-policy/off-policy axis, and why that single difference changes everything about convergence guarantees.

Master the mathematical skeleton of reinforcement learning. Learn how the agent-environment loop formalizes decision-making, why the Markov property is the key assumption, and how Bellman expectation equations link policies to value functions.

When you know the full MDP model, dynamic programming finds the optimal policy exactly. Learn the Bellman optimality equation, the contraction argument that guarantees convergence, and the concrete difference between policy iteration and value iteration — with a value-iteration code walkthrough.