Skip to main content
1 of 3
AD1984
  • 155
  • 1
  • 8

Functional Taylor expansion for differential entropy

Consider an continuous distribution $F$ with density $f$. The (differential) Shannon entropy of $f$ is

$h(f)=-\int f(x)\log f(x) dx$.

In the literature of differential entropy estimation, oftentimes analyzing the performance of an estimator relies on a functional Taylor expansion (sometimes termed `Von Mises Expansion'). For two densities $f$ and $g$, the expansion of $h(f)$ about $g$ reads as (see, e.g., https://www.cs.cmu.edu/~aarti/Class/10704_Spring15/lecs/lec5.pdf)

$h(f)=h(g)+\int \big(\log g(x) +1\big)\big(g(x)-f(x)\big)dx + O\big(||f-g||_2^2\big)$

However, I couldn't fully figure out under what assumptions on $f$ is this expansion valid. Would very much appreciate a clarification on what are the minimal assumptions on $f$ for the above to hold true. Thanks!

AD1984
  • 155
  • 1
  • 8