KL Is Leftover Surprise Between Guesses

Truth 0.8, guess 0.5: entropy 0.5004, cross-entropy 0.6931, KL 0.1927 nats. The reverse is 0.2231. Perfect loss is the entropy, not zero.

This Probability lesson is part of a paid plan. Sign in with a plan that includes this course to read it.