Information Theory and Compression

Shannon proved in 1948 that information has a hard, measurable limit, and nearly every file, stream and disk you touch is built on that result. This path works through it. You will learn what entropy really measures and why cross-entropy is the loss function that trains language models, how Huffman and arithmetic coding approach the compression floor, where lossy formats like JPEG and AAC actually discard information and why that is a deliberate tradeoff, and how error-correcting codes let data survive a noisy channel.