All lessons
cross-entropy
2 free lessons tagged cross-entropy across Computer Science, AI. Each one is a short sequence of focused steps with narration and a five-question quiz at the end — take them in any order, no signup required.
Computer Science
intermediateEntropy: Measuring Information in Bits
Shannon's entropy measures the average surprise of a source, in bits, and it sets a hard floor on compression. Build it up from surprisal through joint and conditional entropy, mutual information, KL divergence and cross-entropy, with every number worked out by hand.
10 steps·~15 min
AI
intermediateLLM Pretraining: Data, Loss, and What Actually Happens
A deep dive into how large language models learn from raw text: the next-token prediction objective, cross-entropy loss, the messy reality of web data curation (Common Crawl, dedup, quality filters), and the lineage from The Pile to FineWeb.
12 steps·~18 min