# Obtaining the error term of binomial distribution's entropy from the differential entropy of a Gaussian distribution

It is known that the first order error term in the Shannon entropy formula for a binomial distribution is $$1/n$$ (for example, see the Wikipedia page Binomial distribution), where in the limit $$n \to \infty$$ the entropy of a binomial approaches the Gaussian one. There are quite technical papers which have calculated the entropy of a binomial distribution exactly; however, I would like to see this error correction $$1/n$$ to the Gaussian in a simpler way. So, I do the following.

Here, in Equation (4), the binomial distribution in the limit $$n \to \infty$$ has been written as $$f(x) = \frac{1}{\sqrt{2\pi npq}} e^{- (x - np)^2/2npq} \bigg[1+\mathcal{O}\Big(\frac{1}{\sqrt{n}}\Big)\bigg].$$.

In order the limit $$n \to \infty$$ to be well-defined and $$f(x)$$ doesn't vanish, we standardize $$f(x)$$ by defining $$\mu \equiv np$$, $$\sigma \equiv \sqrt{npq}$$, and $$g(x) \equiv \sigma f(\sigma x+\mu)$$, then we have: $$g(x) = \frac{1}{\sqrt{2\pi}} e^{- x^2/2} \bigg[1+\mathcal{O}\Big(\frac{1}{\sqrt{n}}\Big)\bigg],$$ where the term outside the bracket is a Gaussian with mean $$0$$ and standard deviation $$1$$.

Now, we can calculate the differential Shannon entropy of $$g(x)$$ as $$S = -\int dx \, g(x) \ln g(x)$$, which reads $$S = \frac{1}{2} \ln 2 \pi e + \mathcal{O}\Big(\frac{1}{\sqrt{n}}\Big).$$

The first term is correctly the entropy of the standard Gaussian but the error is of the order $$1 / \sqrt{n}$$. If we assume the result quoted in Wikipedia is correct, what is wrong with my approach?

Edit 1 (Comment on answer 1)

By the method of moment-generating functions, one can show that the standardized binomial distribution, $$B(n, p)$$, approaches the moment-generating function of the standard Gaussian, $$\mathcal{N}(0,1)$$, when $$n \to \infty$$. Thus, one expects the entropy of the former approaches $$(1/2) \ln 2 \pi e$$, as $$n \to \infty$$, I'm looking for the error terms of this expansion.

• the order $1/\sqrt n$ correction is multiplied by a function that is odd in $x-\mu$, which is why the contribution to the entropy vanishes to that order, and the first nonzero correction is of order $1/n$. Jul 24 at 20:25

## 1 Answer

The confusion about $$1/\sqrt n$$ versus $$1/n$$ corrections is addressed below. First, for reference, let me quote the relevant results, all following from this source.

The Gaussian entropy is $$S_0=\tfrac{1}{2}\ln(2\pi \sigma^2)+\tfrac{1}{2}$$, the leading correction $$\delta S$$ is of order $$1/n$$ and depends on the distribution.

• For the binomial distribution, with variance $$\sigma^2=npq$$ one has $$\delta S=\frac{1}{3n}-\frac{1}{12\sigma^2}.$$ This vanishes if $$p=q=1/2$$, the leading order correction is then of order $$1/n^2$$.
• For the Poisson distribution, with variance $$\sigma^2=n\lambda$$ one has $$\delta S=-\frac{1}{12\sigma^2}.$$
• For the negative binomial distribution, with variance $$\sigma^2=nqp^{-2}$$ one has $$\delta S =\frac{1}{6n}-\frac{1}{12\sigma^2}.$$

More generally, for a discrete distribution with variance $$\sigma^2=n\kappa_2$$ and third cumulant $$n\kappa_3$$ one has $$\delta S=-\frac{\kappa_3^2}{12\kappa_2^3 n}.$$ If the third cumulant vanishes the correction becomes of higher order in $$1/n$$.

Let me try to make contact with the calculation in the OP, to see where the confusion arises. I rescale $$z=(x-np)/\sqrt{npq}$$ to write the expansion $$f(x) = \frac{1}{\sqrt{2\pi npq}} e^{- (x - np)^2/2npq} \bigg[1+\mathcal{O}\Big(\frac{1}{\sqrt{n}}\Big)\bigg]$$ of the binomial distribution around the Gaussian in the form $$\tilde{f}(z)=\frac{1}{\sqrt{2\pi npq}} e^{-z^2/2}\bigg[1+h(z)n^{-1/2}\bigg].$$ In the large-$$n$$ limit, the sum over the discrete variable $$x$$ may be replaced by $$\sqrt{npq}\int_{-\infty}^\infty dz$$. We thus find the entropy $$S=-\sqrt{npq}\int_{-\infty}^\infty \tilde{f}(z)\ln \tilde{f}(z)\,dz.$$

Now comes the key point: the function $$h(z)$$ is odd in $$z$$ (see eq. 4.3 in the cited reference). The order $$1/\sqrt n$$ correction then vanishes since it is an integral over an odd function, and the next term of order $$1/n$$ is the first nonzero correction.