Properties of the binary entropy function

Here are some properties of the binary entropy function (bits) courtesy of Sergio Verdú. Please contribute any others you have found useful.

${0 \leq h(p) \leq 1\phantom{\frac{p}{p}}}$
${h(0.5) = 1 \mbox{~bit} \phantom{\frac{p}{p}}}$
${ h(p) = h(1-p) \phantom{\frac{p}{p}}}$
${h(0.11003) = 0.5 \mbox{~bits} \phantom{\frac{p}{p}}}$
${h \left( \frac{1}{k} \right) = \log_2 k - \left(1 - \frac{1}{k} \right) \log_2 (k-1), ~~k> 1}$
${\frac{d h(p)}{dp}|_{p=0} = \infty \phantom{\frac{p}{p}}}$
${\frac{d h(p)}{dp} = \log_2 \frac{1-p}{p}, ~~ 0 < p < 1}$
${\frac{d}{dp} \frac{h(p)}{1-p} = \frac{1}{(1-p)^2 }\log_2 \frac{1}{p} }$
${ \frac{h(p)}{1-p} ~\mbox{is monotonically increasing on} ~(0,1)}$
${p \log \alpha - h(p) ~\mbox{is monotonically decreasing/increasing on} }$ ${~\left(0, \frac{1}{1 + \alpha}\right) ~\mbox{and}~\left(\frac{1}{1 + \alpha}, 1 \right) ~ \mbox{respectively, for any}~\alpha > 0}$
${h(p) ~\mbox{is concave on} ~(0,1) \phantom{\frac{p}{p}}}$
${h(p) = p \log_2 \frac{e}{p} - \frac{\log_2 e}{2} p^2 + o(p^2) \phantom{\frac{p}{p}}}$
${\lim_{\alpha \downarrow 0} \frac{1}{\alpha} h \left( \frac{1}{1+\alpha} \right) - \log_2 \left( 1 + \frac{1}{\alpha} \right) = \log_2 e}$
${h \left( \frac{1+x}{2} \right) = 1 - \sum_{k=1}^\infty \frac{x^{2k}}{(2k-1) 2k} \log_2 e , ~~| x | \leq 1}$
${\lim_{n \rightarrow \infty} \frac{1}{n} \log_2 \binom{n}{k_n} = h (p ) ~\Longleftarrow ~\lim_{n \rightarrow \infty} \frac{k_n}{n} = p}$
${(1 - p q ) h \left( \frac{p - p q}{1 - p q } \right) = h(p) + p h(q) - h(pq) ~~0\leq p \leq 1, 0\leq q \leq 1, pq < 1 }$
${h(p) \leq 2 \sqrt{p (1-p)}\phantom{\frac{p}{p}}}$
${q h (p) \leq h ( q p ) , ~(p,q) \in [0,1]^2 \phantom{\frac{p}{p}}}$
${\int_0^{1} h ( x ) \, d x = \frac{1}{2} \log_2 {e}}$
${\int_0^1 \frac{h \left(x^\alpha\right)}{x} \,dx = \frac{\pi^2}{6 \alpha}\log_2 {e} }$
${\int_0^{\frac{\pi}{2}} h ( \sin^2 \alpha ) \, d\alpha = \frac{\pi}{2} \log_2 \frac{4}{e}}$
${\int_0^{1} u^{2k} h \left( \frac{1-u}{2}\right) \, du = \frac{\log_2 {e}}{(2k+1)(2k+2)} \sum_{i=0}^k \frac{1}{2i+1}~~~k=0,1,2, \ldots}$
${\max_{0 \leq \alpha \leq 1} h(\alpha ) + (1- \alpha) \beta_0 + \alpha \beta_1 = \log_2 ( 2^{\beta_0} + 2^{\beta_1} )}$
${h ( h^{-1} ( x ) * p ) ~\mbox{is convex on}~ 0<x<1 ~\mbox{for any}~ 0 \leq p \leq 1, \mbox{~where}}$ ${ ~ a * p = a (1-p) + p (1-a) \mbox{~and}~ h^{-1} ~\mbox{is the inverse of}~ h( x ) ~\mbox{on}~ \left[0, 1/2 \right] }$

2 thoughts on “Properties of the binary entropy function”

Sanket Satpathy on November 6, 2012 at 7:38 pm said:

$\max_p\frac{h(p)}{1+p}=\varphi^{-2}$ , where $\varphi=\frac{1+\sqrt{5}}{2}$ is the golden ratio

Reply ↓
- Sergio Verdu on November 9, 2012 at 6:44 pm said:
  
  it’s not the max but the argmax

Information Theory b-log

Properties of the binary entropy function

2 thoughts on “Properties of the binary entropy function”

Leave a Reply Cancel reply