concept

Shannon entropy

A measure of the average information a source produces per symbol - the uncertainty each symbol resolves. Written H, computed as the sum of -p·log₂(p) over all possibilities. A predictable source carries little information per symbol; an unpredictable one carries much. English, thanks to its redundancy and rules, holds only about 1 to 1.5 bits of entropy per letter - far below the 100+ keys on a keyboard. Entropy tells you how far a message can be compressed without loss.