It is known that the first order error term in the Shannon entropy formula for a binomial distribution is $1/n$ (for example, see the Wikipedia page Binomial distribution), where in the limit $n \to \infty$ the entropy of a binomial approaches the Gaussian one. There are quite technical papers which have calculated the entropy of a binomial distribution exactly; however, I would like to see this error correction $1/n$ to the Gaussian in a simpler way. So, I do the following.

Here, in Equation (4), the binomial distribution in the limit $n \to \infty$ has been written as $$f(x) = \frac{1}{\sqrt{2\pi npq}} e^{- (x - np)^2/2npq} \bigg[1+\mathcal{O}\Big(\frac{1}{\sqrt{n}}\Big)\bigg].$$.

In order the limit $n \to \infty$ to be well-defined and $f(x)$ doesn't vanish, we standardize $f(x)$ by defining $\mu \equiv np$, $\sigma \equiv \sqrt{npq}$, and $g(x) \equiv \sigma f(\sigma x+\mu)$, then we have: $$g(x) = \frac{1}{\sqrt{2\pi}} e^{- x^2/2} \bigg[1+\mathcal{O}\Big(\frac{1}{\sqrt{n}}\Big)\bigg],$$ where the term outside the bracket is a Gaussian with mean $0$ and standard deviation $1$.

Now, we can calculate the differential Shannon entropy of $g(x)$ as $S = -\int dx \, g(x) \ln g(x)$, which reads $$S = \frac{1}{2} \ln 2 \pi e + \mathcal{O}\Big(\frac{1}{\sqrt{n}}\Big).$$

The first term is correctly the entropy of the standard Gaussian but the error is of the order $1 / \sqrt{n}$. If we assume the result quoted in Wikipedia is correct, what is wrong with my approach?

Edit 1 (Comment on answer 1)

By the method of moment-generating functions, one can show that the standardized binomial distribution, $B(n, p)$, approaches the moment-generating function of the standard Gaussian, $\mathcal{N}(0,1)$, when $n \to \infty$. Thus, one expects the entropy of the former approaches $(1/2) \ln 2 \pi e$, as $n \to \infty$, I'm looking for the error terms of this expansion.

  • $\begingroup$ the order $1/\sqrt n$ correction is multiplied by a function that is odd in $x-\mu$, which is why the contribution to the entropy vanishes to that order, and the first nonzero correction is of order $1/n$. $\endgroup$ Jul 24, 2021 at 20:25

1 Answer 1


The confusion about $1/\sqrt n$ versus $1/n$ corrections is addressed below. First, for reference, let me quote the relevant results, all following from this source.

The Gaussian entropy is $S_0=\tfrac{1}{2}\ln(2\pi \sigma^2)+\tfrac{1}{2}$, the leading correction $\delta S$ is of order $1/n$ and depends on the distribution.

• For the binomial distribution, with variance $\sigma^2=npq$ one has $\delta S=\frac{1}{3n}-\frac{1}{12\sigma^2}.$ This vanishes if $p=q=1/2$, the leading order correction is then of order $1/n^2$.
• For the Poisson distribution, with variance $\sigma^2=n\lambda$ one has $\delta S=-\frac{1}{12\sigma^2}.$
• For the negative binomial distribution, with variance $\sigma^2=nqp^{-2}$ one has $\delta S =\frac{1}{6n}-\frac{1}{12\sigma^2}.$

More generally, for a discrete distribution with variance $\sigma^2=n\kappa_2$ and third cumulant $n\kappa_3$ one has $$\delta S=-\frac{\kappa_3^2}{12\kappa_2^3 n}.$$ If the third cumulant vanishes the correction becomes of higher order in $1/n$.

Let me try to make contact with the calculation in the OP, to see where the confusion arises. I rescale $z=(x-np)/\sqrt{npq}$ to write the expansion $$f(x) = \frac{1}{\sqrt{2\pi npq}} e^{- (x - np)^2/2npq} \bigg[1+\mathcal{O}\Big(\frac{1}{\sqrt{n}}\Big)\bigg]$$ of the binomial distribution around the Gaussian in the form $$\tilde{f}(z)=\frac{1}{\sqrt{2\pi npq}} e^{-z^2/2}\bigg[1+h(z)n^{-1/2}\bigg].$$ In the large-$n$ limit, the sum over the discrete variable $x$ may be replaced by $\sqrt{npq}\int_{-\infty}^\infty dz$. We thus find the entropy $$S=-\sqrt{npq}\int_{-\infty}^\infty \tilde{f}(z)\ln \tilde{f}(z)\,dz.$$

Now comes the key point: the function $h(z)$ is odd in $z$ (see eq. 4.3 in the cited reference). The order $1/\sqrt n$ correction then vanishes since it is an integral over an odd function, and the next term of order $1/n$ is the first nonzero correction.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.