the-pile

1 free lesson tagged the-pile across AI. Each one is a short sequence of focused steps with narration and a five-question quiz at the end — take them in any order, no signup required.

intermediate

LLM Pretraining: Data, Loss, and What Actually Happens

A deep dive into how large language models learn from raw text: the next-token prediction objective, cross-entropy loss, the messy reality of web data curation (Common Crawl, dedup, quality filters), and the lineage from The Pile to FineWeb.

12 steps·~18 min

LLM Pretraining: Data, Loss, and What Actually Happens

Related topics