Binary entropy function

Entropy of a Bernoulli trial as a function of success probability, called the binary entropy function.

The binary entropy function, usually denoted $H_{2}(p)$ or $H_{b}(p)$, is defined as the entropy of a Bernoulli trial with probability of success p. Mathematically, the Bernoulli trial is modelled as a random variable X that can take on only two values: 0 and 1. The event $X=1$ is considered a success and the event $X=0$ is considered a failure. (These two events are mutually exclusive and exhaustive.)

If $\mathrm {Pr} (X=1)=p,$ then $\mathrm {Pr} (X=0)=1-p$ and the entropy of X is given by

$H_{2}(p)=H(X)=-p\log p-(1-p)\log(1-p).\,$

The logarithms in this formula are usually taken (as shown in the graph) to the base 2. See binary logarithm.

When $p={\frac {1}{2}},$ the binary entropy function attains its maximum value. This is the case of the unbiased bit, the most common unit of information.

See also

• Information theory
• Information entropy