evaluation

If your cost per round trip equals the move you are trying to capture, you need 100 percent directional accuracy to break even. This lesson computes that hurdle, replaces accuracy with metrics tied to a tradeable decision, and covers the capacity and latency limits that decide whether a real edge is worth anything.

At a one-event horizon, 92 percent of mid-price labels are exactly no-change, so a model that always predicts flat scores 92 percent accuracy. This lesson computes that baseline across horizons and works through the four mechanisms that turn a genuine measurement into a number nobody can reproduce.

A deployed recommender chooses its own future training data: it shows items, users respond to what was shown, and those responses become the next model's ground truth. This lesson maps the loop's consequences, exposure bias, popularity compounding, narrowing candidate pools, explains why offline metrics reward imitation of the loop, and covers the exploration budget that keeps the system learning.

Search quality arguments end when there is a number, and begin again over whether the number is honest. This lesson builds offline evaluation from its atoms: a judgment set, precision and recall at k, MRR for known-item queries, and NDCG computed by hand for graded relevance. Then the online half: clicks, position bias, and why the top result gets clicked even when it is wrong.

An aggregate metric tells you a model is worse than you hoped and nothing about why. Evaluation that writes results back onto each sample turns a number into a set of images you can look at. This lesson covers the evaluation methods and their protocols, per-sample true and false positive counts, how a confusion matrix cell becomes a view, and where this workflow stops.

The exchange is a case study with transferable rules. A conclusion resting on failures needs a failure taxonomy. Every instance must be verified solvable before anyone is scored against it. Output format is a confound whenever answers get long. And when two explanations fit the same data, the productive move is to find the prediction on which they differ, then test it.

The response disputed none of the data and argued the experiment measured something other than reasoning. Models had to print move lists exceeding their output limits, and said so in the transcripts. Some instances had no solution and were scored as failures anyway. And asking for a program instead of a move list produced high accuracy on instances reported as total collapse.

Apple researchers built an evaluation designed to fix a real problem with benchmarks: puzzles where difficulty turns up smoothly while the logic stays identical, and every step can be checked. They found accuracy collapsing to zero past a threshold, and not improving when the solution algorithm was handed to the model. This lesson covers the design and why it was a good one.

A model went from 27 percent to around 57 percent, so it is more than halfway to AGI and the rest arrives shortly. That inference is wrong in at least four ways, and working through why is more useful than the score itself. This lesson covers the linearity assumption, construct validity, contamination, and what the framework is good for once you stop reading it as a progress bar.

The framework scores ten cognitive domains at ten percent each. Running it produces something more useful than the headline totals of 27 percent for GPT-4 and around 57 for GPT-5: a jagged profile, where a model is at or near full marks on some domains and at zero on others. This lesson walks the domains, reads both profiles column by column, and shows what the jaggedness explains.

For a term that anchors company charters, safety policy and hundreds of billions in investment, AGI has been remarkably undefined. A 2025 paper with over thirty authors proposes fixing that by borrowing psychology's most validated framework for human intelligence. This lesson covers why the definition was missing, what the paper anchors to, and what the borrowing assumes.

Training on your own model's output degrades it, and the widely cited result rests on an assumption people rarely notice: that synthetic data replaces real data. Follow-up work showed accumulating synthetic alongside real bounds the error instead. This lesson covers both stages of collapse, why the rule is accumulate rather than replace, how to set a mixing ratio, and the evaluation discipline.

Generation is the cheap half. What you discard determines what the student learns. This lesson orders the filters by strength: machine verification where an answer can be checked, self-consistency where it cannot, LLM-as-judge with its known position and length biases, and cheap heuristics. It ends on contamination, the failure that invalidates results rather than degrading them.

Constrained decoding guarantees syntax and can degrade reasoning. Published work found strict formats hurt reasoning tasks while helping classification, because a schema demanding the answer first denies the model room to derive it. This lesson covers where the damage comes from, why field order is thinking order, the enum trap that manufactures confident errors, and how to measure the cost.

Quantization damage does not show up where people look for it. Perplexity barely moves while hard tasks degrade, and long reasoning suffers most because error compounds. This lesson covers building an evaluation that detects real loss, where the published cliffs are, KV cache quantization as a separate lever, end-to-end memory sizing, and the rollout that catches what evals miss.

The most prominent study in this area found a language model predicting stock reactions from headlines, and it is usually reported as proving something it explicitly does not claim. This lesson reads it precisely, follows its qualifiers to their consequences, and shows why a real statistical result and a tradable strategy are different things.

Backtesting a language model signal has a defect no other signal has. The model was trained on text from the period being tested, so it may already know what happened next, and it uses that knowledge even when instructed not to. This lesson establishes the problem from the published evidence, shows why prompting does not fix it, and covers what does.

Displacement error is the field's default metric and it rewards the wrong behaviour: hedged average predictions, physically impossible trajectories, and best-of-many reporting that flatters diverse nonsense. What to measure instead.

Where discrete diffusion goes after language: unified multimodal models, sequence design in biology, the fixed-length problem nobody has cleanly solved, and why alignment and evaluation both need rebuilding.

Every funder asks for impact and few organisations can afford real evaluation. This lesson covers what you can honestly claim, the biases that make feedback flattering, where AI genuinely helps with qualitative data at scale, and why generating impact claims is the one thing that ends an organisation.

A release gate turns evaluation results into a decision someone can defend. This lesson covers what belongs in the gate, setting thresholds before you see results, blocking versus flagging, how to gate a provider model change you did not initiate, and running the gate so it stays credible rather than becoming a formality.

Software QA assumes a specification, deterministic behaviour, and a system that changes only when you change it. AI features violate all three. This lesson covers what actually breaks, why test counts and pass rates stop meaning anything, and the shift from verifying correctness to characterising a distribution of behaviour.

Since a model cannot judge its own output, detection has to compare it against something external. This lesson covers grounding through retrieval and why it reduces rather than eliminates the problem, then the detection methods that work: self-consistency sampling, entailment checking against sources, claim decomposition, and chain-of-verification.

Hallucination is not a bug that will be patched out. This lesson covers why a next-token predictor produces confident falsehoods, the dual-axis taxonomy separating intrinsic from extrinsic and factuality from faithfulness, why fluency carries no signal about truth, and the theoretical result that the problem cannot be fully eliminated.

Guardrails you have not attacked are guardrails you have only hoped about. This lesson covers manual and automated red-teaming: the attack methods worth knowing, how to run an exercise that produces findings rather than reassurance, what to measure, and how to turn results into regression tests that survive the next model update.

Vendor claims are marketing until you test them. This lesson covers the diligence pack worth requesting, why the instructions for use are the single most valuable document in a high-risk purchase, questions about model provenance and the AI-specific attack surface, designing a pilot that can actually fail, shadow-mode evaluation, and the red flags that should end a procurement early.

Retrieval solved the read side, so the constraint moved to what gets written down. This lesson covers the capture write path: architecture decision records, docs-as-code, change-triggered updates, and making capture cheap enough to survive. Then how to evaluate a company brain with golden questions, groundedness, and a permission regression suite, plus the failure modes that end these projects.

Turning a voice AI prototype into a real product means facing hard choices. Learn the two competing architectures (the debuggable cascaded pipeline versus end-to-end speech-to-speech models), how function calling lets an agent actually do things, why cost and observability drive real decisions, the failure modes that break voice agents in production, and how to evaluate whether one actually works.

Learn how to systematically evaluate Retrieval-Augmented Generation systems in production using RAGAS, TruLens, and Phoenix — covering golden sets, retrieval drift, embedding drift, and cost-aware eval scheduling.

A rigorous look at how to measure agent performance — trajectory-level vs final-answer evals, canonical multi-step benchmarks (SWE-bench, WebArena, OSWorld, GAIA), LLM-as-judge pitfalls, and why your eval is your spec.

A rigorous guide to measuring RAG quality using RAGAS metrics — faithfulness, answer relevancy, context precision, and context recall — plus how to build a golden dataset and recognize where automated metrics fall short.

A field guide to the LLM benchmarks practitioners cite in 2026 — what each one measures, where it's saturated, where contamination risk is high, and why benchmark gains rarely transfer to your task without your own eval.

A practical decision framework for picking an LLM in 2026 — define the task, build an offline eval, measure quality + latency + cost on real candidates, and avoid the classic trap of optimizing for benchmarks instead of your task.

A capability-by-capability tour of frontier LLMs in 2026 — which models are strong at reasoning, code, math, long-context, multilingual, multimodal, and tool use, with hedged comparisons instead of point-estimate benchmark wars.

Dive into LangSmith, the developer platform for building and evaluating robust Large Language Model (LLM) applications. Learn how to trace execution paths, debug complex chains, and rigorously evaluate your LLM's performance to ensure reliability and quality.