pretraining

2 free lessons tagged pretraining across AI. Each one is a short sequence of focused steps with narration and a five-question quiz at the end — take them in any order, no signup required.

advanced

Training Diffusion Language Models at Scale

What it takes to train a diffusion language model to billions of parameters: the from-scratch recipe, adaptation from an autoregressive checkpoint, the supervision economics, and the capability that bidirectional training gets for free.

9 steps·~14 min

intermediate

LLM Pretraining: Data, Loss, and What Actually Happens

A deep dive into how large language models learn from raw text: the next-token prediction objective, cross-entropy loss, the messy reality of web data curation (Common Crawl, dedup, quality filters), and the lineage from The Pile to FineWeb.

12 steps·~18 min

Training Diffusion Language Models at Scale

LLM Pretraining: Data, Loss, and What Actually Happens

Related topics