L3-08 · Entropy and uncertainty

Learn¶

Entropy measures uncertainty in a probability distribution. It does not measure the size or importance of an outcome. A fair coin has more entropy than a coin that always gives heads. The logarithm base determines the information unit. Base two gives bits.

For a discrete distribution, multiply each probability by the negative logarithm of that probability, then add. A zero-probability term contributes zero by continuity. For two possible outcomes, entropy is largest when their probabilities are equal. It becomes zero when either outcome is certain.

Classical entropy applies to recorded outcomes. Quantum states use a related quantity called von Neumann entropy, which depends on the density matrix eigenvalues. A pure quantum state can have zero von Neumann entropy while a chosen measurement on that state still has uncertain outcomes.

Practice¶

Check your understanding¶

Which binary distribution has the largest entropy?

• A. Probabilities 0.5 and 0.5

• B. Probabilities 0.9 and 0.1

• C. Probabilities 1 and 0

Sources¶

This lesson draws on Watrous, 2018Preskill, n.d..

Next: L3-09 · Optimization and learning