testing

A benchmark is an experiment, and most are badly designed enough to produce confident wrong answers. This lesson covers what a measurement must control to mean anything, the ways microbenchmarks lie including code the compiler deletes, how to catch regressions in continuous integration despite noisy machines, and how to recognise the point where optimising stops paying.

A test suite is a product with a maintenance cost, and two forces decide whether it stays useful: flakiness, which destroys the signal, and the metrics teams use to judge it, which mostly measure the wrong thing. This lesson covers where flakiness comes from, the arithmetic that makes it fatal at scale, what coverage research actually found, and what to measure instead.

An example-based test checks the cases you thought of, which is exactly the set that excludes your bugs. Property-based testing inverts it: state a law the code must obey for all inputs, let the machine hunt for a counterexample, and let it shrink that counterexample to something you can read. This lesson covers the property patterns, generators, shrinking, and where the technique stops fitting.

Replacing a real dependency with a stand-in is what makes a unit test fast, deterministic, and able to reach states you cannot otherwise produce. It is also how a suite ends up green while the system is broken. This lesson covers the five kinds of double, when each is right, and the two failure modes that follow every team who reaches for mocks by reflex.

Every test costs time to write, time to run, and time to maintain when the code moves. This lesson works out what that budget should buy: what each level of test can and cannot catch, why the pyramid has the shape it does, why some teams invert it, and the arithmetic that decides both questions before anyone argues about it.

Equality fails on floats, and both standard replacements fail too: absolute tolerance breaks at scale, relative tolerance breaks near zero. This lesson works through why each fails, what the combined form actually does, and how to choose a tolerance from the problem rather than copying a magic constant.

An AI QA function has to be built while features are already shipping. This lesson covers the order of work that produces something useful fast, where the function should sit, how to handle a team that resists a new gate, what to measure about the function itself, and the failure modes that quietly end it.

A release gate turns evaluation results into a decision someone can defend. This lesson covers what belongs in the gate, setting thresholds before you see results, blocking versus flagging, how to gate a provider model change you did not initiate, and running the gate so it stays credible rather than becoming a formality.

Software QA assumes a specification, deterministic behaviour, and a system that changes only when you change it. AI features violate all three. This lesson covers what actually breaks, why test counts and pass rates stop meaning anything, and the shift from verifying correctness to characterising a distribution of behaviour.

A rigorous look at how to measure agent performance — trajectory-level vs final-answer evals, canonical multi-step benchmarks (SWE-bench, WebArena, OSWorld, GAIA), LLM-as-judge pitfalls, and why your eval is your spec.

Why traditional software testing falls apart on LLMs, the four evaluation regimes that work in practice (golden sets, LLM-as-judge, human review, online metrics), and how to wire them together without drowning in ungrounded scores.