ai-safety

The paper's answer to safety is that reward signals adapt to human feedback: a network inside the reward function learns which grounded signals to weight from how people respond. This lesson works through that proposal and the published critique arguing it fails twice over, on specification gaming and on goal misgeneralization, and what an honest reading of the disagreement leaves you with.

What happens when a prediction model leaves the benchmark: adversarial fragility, distribution shift, the feedback loop where your own robot changes the behaviour it predicts, and how planners consume uncertainty safely.

Where discrete diffusion goes after language: unified multimodal models, sequence design in biology, the fixed-length problem nobody has cleanly solved, and why alignment and evaluation both need rebuilding.

Turning interpretability stories into tested claims: activation patching, model editing, steering vectors, the attribution graphs that trace circuits in production models, and the problems the field has not solved.

How an unsupervised model trained on activations pulls features back out of superposition, what it found when scaled to a production language model, and where the method still breaks down.

Why individual neurons are the wrong unit for understanding a neural network, and what the superposition hypothesis says is really going on inside the activations.

Hallucination is not a bug that will be patched out. This lesson covers why a next-token predictor produces confident falsehoods, the dual-axis taxonomy separating intrinsic from extrinsic and factuality from faithfulness, why fluency carries no signal about truth, and the theoretical result that the problem cannot be fully eliminated.

Guardrails you have not attacked are guardrails you have only hoped about. This lesson covers manual and automated red-teaming: the attack methods worth knowing, how to run an exercise that produces findings rather than reassurance, what to measure, and how to turn results into regression tests that survive the next model update.

Guardrails are a layered control system around a model that cannot police itself. This lesson covers the rail types, rules versus classifiers versus model-based judges, the tools that implement them, the latency and false-positive budget that constrains every design, and why architectural containment beats filtering.

Guardrails exist because a language model cannot reliably distinguish instructions from data. This lesson builds the threat model: the OWASP Top 10 for LLM Applications, direct and indirect prompt injection, jailbreaks, data leakage, and why tool-using agents turn a content problem into a security one.

Why neurons are polysemantic, how the superposition hypothesis explains it, and how sparse autoencoders use dictionary learning to pull a model's activations apart into monosemantic, steerable features, plus the failure modes and the top-k and gated fixes.