Obtaining the error term of binomial distribution's entropy from the differential entropy of a Gaussian distribution

Question

It is known that the first order error term in the Shannon entropy formula for a binomial distribution is $1/n$ (for example, see the Wikipedia page Binomial distribution), where in the limit $n \to \infty$ the entropy of a binomial approaches the Gaussian one. There are quite technical papers which have calculated the entropy of a binomial distribution exactly; however, I would like to see this error correction $1/n$ to the Gaussian in a simpler way. So, I do the following.

Here, in Equation (4), the binomial distribution in the limit $n \to \infty$ has been written as $$f(x) = \frac{1}{\sqrt{2\pi npq}} e^{- (x - np)^2/2npq} \bigg[1+\mathcal{O}\Big(\frac{1}{\sqrt{n}}\Big)\bigg].$$.

In order the limit $n \to \infty$ to be well-defined and $f(x)$ doesn't vanish, we standardize $f(x)$ by defining $\mu \equiv np$, $\sigma \equiv \sqrt{npq}$, and $g(x) \equiv \sigma f(\sigma x+\mu)$, then we have: $$g(x) = \frac{1}{\sqrt{2\pi}} e^{- x^2/2} \bigg[1+\mathcal{O}\Big(\frac{1}{\sqrt{n}}\Big)\bigg],$$ where the term outside the bracket is a Gaussian with mean $0$ and standard deviation $1$.

Now, we can calculate the differential Shannon entropy of $g(x)$ as $S = -\int dx \, g(x) \ln g(x)$, which reads $$S = \frac{1}{2} \ln 2 \pi e + \mathcal{O}\Big(\frac{1}{\sqrt{n}}\Big).$$

The first term is correctly the entropy of the standard Gaussian but the error is of the order $1 / \sqrt{n}$. If we assume the result quoted in Wikipedia is correct, what is wrong with my approach?

Edit 1 (Comment on answer 1)

By the method of moment-generating functions, one can show that the standardized binomial distribution, $B(n, p)$, approaches the moment-generating function of the standard Gaussian, $\mathcal{N}(0,1)$, when $n \to \infty$. Thus, one expects the entropy of the former approaches $(1/2) \ln 2 \pi e$, as $n \to \infty$, I'm looking for the error terms of this expansion.

Answer 1

The confusion about $1/\sqrt n$ versus $1/n$ corrections is addressed below. First, for reference, let me quote the relevant results, all following from this source.

The Gaussian entropy is $S_0=\tfrac{1}{2}\ln(2\pi \sigma^2)+\tfrac{1}{2}$, the leading correction $\delta S$ is of order $1/n$ and depends on the distribution.

• For the binomial distribution, with variance $\sigma^2=npq$ one has $\delta S=\frac{1}{3n}-\frac{1}{12\sigma^2}.$ This vanishes if $p=q=1/2$, the leading order correction is then of order $1/n^2$.
• For the Poisson distribution, with variance $\sigma^2=n\lambda$ one has $\delta S=-\frac{1}{12\sigma^2}.$
• For the negative binomial distribution, with variance $\sigma^2=nqp^{-2}$ one has $\delta S =\frac{1}{6n}-\frac{1}{12\sigma^2}.$

More generally, for a discrete distribution with variance $\sigma^2=n\kappa_2$ and third cumulant $n\kappa_3$ one has $$\delta S=-\frac{\kappa_3^2}{12\kappa_2^3 n}.$$ If the third cumulant vanishes the correction becomes of higher order in $1/n$.

---

Let me try to make contact with the calculation in the OP, to see where the confusion arises. I rescale $z=(x-np)/\sqrt{npq}$ to write the expansion $$f(x) = \frac{1}{\sqrt{2\pi npq}} e^{- (x - np)^2/2npq} \bigg[1+\mathcal{O}\Big(\frac{1}{\sqrt{n}}\Big)\bigg]$$ of the binomial distribution around the Gaussian in the form $$\tilde{f}(z)=\frac{1}{\sqrt{2\pi npq}} e^{-z^2/2}\bigg[1+h(z)n^{-1/2}\bigg].$$ In the large-$n$ limit, the sum over the discrete variable $x$ may be replaced by $\sqrt{npq}\int_{-\infty}^\infty dz$. We thus find the entropy $$S=-\sqrt{npq}\int_{-\infty}^\infty \tilde{f}(z)\ln \tilde{f}(z)\,dz.$$

Now comes the key point: the function $h(z)$ is odd in $z$ (see eq. 4.3 in the cited reference). The order $1/\sqrt n$ correction then vanishes since it is an integral over an odd function, and the next term of order $1/n$ is the first nonzero correction.