All lessons
swe-bench
3 lessons tagged swe-bench: free, quiz-checked micro-lessons.
AI
intermediateEvaluating AI Agents: From Final Answers to Full Trajectories
A rigorous look at how to measure agent performance — trajectory-level vs final-answer evals, canonical multi-step benchmarks (SWE-bench, WebArena, OSWorld, GAIA), LLM-as-judge pitfalls, and why your eval is your spec.
12 steps·~18 min
Business
advancedCoding Agents: Productivity Studies, Benchmarks, and the METR Slowdown
Hard evidence on AI coding assistants: the Microsoft Research RCTs, METR's surprising 2025 slowdown finding, SWE-bench Verified, and why benchmark scores keep outrunning real-world value.
12 steps·~18 min
AI
intermediateInterpreting LLM Benchmarks: What MMLU, GPQA, and SWE-bench Actually Measure
A field guide to the LLM benchmarks practitioners cite in 2026 — what each one measures, where it's saturated, where contamination risk is high, and why benchmark gains rarely transfer to your task without your own eval.
10 steps·~15 min