mutual-information

1 free lesson tagged mutual-information across Computer Science. Each one is a short sequence of focused steps with narration and a five-question quiz at the end — take them in any order, no signup required.

Computer Science

intermediate

Entropy: Measuring Information in Bits

Shannon's entropy measures the average surprise of a source, in bits, and it sets a hard floor on compression. Build it up from surprisal through joint and conditional entropy, mutual information, KL divergence and cross-entropy, with every number worked out by hand.

10 steps·~15 min

Entropy: Measuring Information in Bits

Related topics