Lessons

Deploying is the moment a working system is replaced by a different one while people are using it. This lesson covers what actually goes wrong at that moment, the research finding that shipping fast and shipping safely are not opposites, and the four deployment strategies as answers to one question: how many users meet a bad version before you find out.

Every deployment strategy assumes you can go back, and that assumption is the one most often false when it matters. This lesson covers what actually makes a rollback work, the database migration pattern that keeps schema changes reversible, the one-way doors that no amount of tooling can undo, and how to tell which kind of change you are about to ship.

A feature flag turns shipping code and exposing behaviour into two independent decisions, which is what lets a team deploy continuously while releasing on someone else's schedule. This lesson covers the four kinds of flag and their very different lifespans, the debt they accumulate, and the discipline that keeps a flag system from becoming untestable.

Two things reliably surprise teams new to AWS: the network they must build before anything can talk, and an invoice driven by charges nobody chose deliberately. The two are connected, because moving data is where much of the cost hides. This lesson covers the virtual network primitives, the traffic charges that follow from them, and how to read a bill.

Cloud storage looks like a long product list and is really three physical shapes, object, block and file, each with an access pattern it is built for and one it is bad at. This lesson covers those three, the difference between durability and availability that people conflate, what eleven nines actually means at scale, and how to choose a database by access pattern.

AWS offers several ways to run code, and they are not competitors so much as points on one spectrum trading control for operational relief. This lesson walks that spectrum from virtual machines to serverless functions, what you hand over at each step, the cold-start and state constraints that decide fit, and the honest cases where serverless is the wrong answer.

AWS offers hundreds of services, which makes it look like a catalogue to memorise. It is not. This lesson gives the four structures everything else hangs from: the physical geography of regions and availability zones, the account as a blast-radius boundary, IAM as the one gatekeeper every call passes, and the responsibility line between you and the provider.

Three services decide whether an agent survives contact with production: what it remembers between sessions, whose authority it acts with when it calls your systems, and whether you can reconstruct what it did after the fact. This lesson covers AgentCore Memory, Identity and Observability, and the delegation problem that makes agent authentication genuinely different.

An agent is only as useful as the things it can do, and connecting many agents to many tools is a multiplication problem that gets expensive fast. This lesson covers what AgentCore Gateway converts into tools and how, the semantic search that stops tool overload, the two directions of authentication, and the managed sandboxes for code and browsing.

An agent is a loop that runs for minutes, holds state, executes code it just wrote, and must not leak anything into the next user's session. This lesson covers what AgentCore Runtime provides that a container does not, why session isolation is the load-bearing guarantee, and where the managed Harness sits against bringing your own loop.

Amazon Bedrock's pitch is that model choice becomes a configuration value instead of a rewrite. This lesson takes that claim apart: the four API dialects Bedrock exposes over the same models, what the unified Converse API actually normalises, what it cannot normalise, and the inference and governance machinery that decides cost and blast radius.

A test suite is a product with a maintenance cost, and two forces decide whether it stays useful: flakiness, which destroys the signal, and the metrics teams use to judge it, which mostly measure the wrong thing. This lesson covers where flakiness comes from, the arithmetic that makes it fatal at scale, what coverage research actually found, and what to measure instead.

An example-based test checks the cases you thought of, which is exactly the set that excludes your bugs. Property-based testing inverts it: state a law the code must obey for all inputs, let the machine hunt for a counterexample, and let it shrink that counterexample to something you can read. This lesson covers the property patterns, generators, shrinking, and where the technique stops fitting.

Replacing a real dependency with a stand-in is what makes a unit test fast, deterministic, and able to reach states you cannot otherwise produce. It is also how a suite ends up green while the system is broken. This lesson covers the five kinds of double, when each is right, and the two failure modes that follow every team who reaches for mocks by reflex.

Every test costs time to write, time to run, and time to maintain when the code moves. This lesson works out what that budget should buy: what each level of test can and cannot catch, why the pyramid has the shape it does, why some teams invert it, and the arithmetic that decides both questions before anyone argues about it.

Two advanced problems decide how much an experimentation platform is actually worth. Variance: most product metrics are so noisy that detecting small effects takes painful sample sizes, and CUPED buys the reduction with data you already have. And interference: in marketplaces and social products the arms affect each other, so the measured difference misstates what full launch will do.

The most natural behaviour in experimentation, checking results daily and stopping when they look significant, quietly destroys the statistical guarantee everyone thinks they have. This lesson shows the peeking mechanism with honest arithmetic, then the fixes: fixed-horizon discipline, group sequential designs, and always-valid inference.

Most wrong experiment results are not statistical subtleties; they are plumbing. This lesson covers how assignment actually works, hashing, not coin flips, why exposure must be logged at the moment of treatment, and the sample ratio mismatch check: the humble comparison of observed to expected group sizes that catches more broken experiments than any other single test.

The companies famous for experimentation did not adopt it out of statistical enthusiasm: they adopted it because their own data showed most confident product ideas fail to improve the metrics they target. This lesson covers why observational product data misleads, what randomisation actually buys, the choice of randomisation unit, and the humbling base rates reported by the teams who measured.

A deployed recommender chooses its own future training data: it shows items, users respond to what was shown, and those responses become the next model's ground truth. This lesson maps the loop's consequences, exposure bias, popularity compounding, narrowing candidate pools, explains why offline metrics reward imitation of the loop, and covers the exploration budget that keeps the system learning.

The ranker is a prediction machine, and the hard question is what it should predict. Clicks are plentiful and poisonous, watch time bends toward length, likes are rare and unrepresentative. This lesson covers implicit feedback, the position bias baked into every training log, multi-objective ranking, and calibration.

Candidate generation's workhorse is the two-tower model: one network embeds users, another embeds items, and relevance becomes a dot product in shared space. This lesson traces the idea from collaborative filtering through matrix factorization to trained towers, why the towers must never mix features, and the cold-start problem geometry alone cannot solve.

A recommender has milliseconds to pick ten items from millions, and no model good enough to rank them all is cheap enough to run on them all. The industry's answer is a funnel: cheap candidate generation cuts millions to hundreds, an expensive ranker orders those hundreds. This lesson builds that architecture, its latency arithmetic, and the multi-source retrieval layer real systems run.

Search quality arguments end when there is a number, and begin again over whether the number is honest. This lesson builds offline evaluation from its atoms: a judgment set, precision and recall at k, MRR for known-item queries, and NDCG computed by hand for graded relevance. Then the online half: clicks, position bias, and why the top result gets clicked even when it is wrong.