2
$\begingroup$

I could not find a really appropriate title for my question (happy to revise) but let me explain.

Suppose $p(x|c)$ is a probability density function over $x \in [0,1]$ which depends continuously on some parameter $c$. Suppose I know that for any positive integer $m$, the entropy functional \begin{align} I^{(m)}[c] = \int dx p^m(x|c) \end{align} is maximized at the same value $c_* $. Does it follow that (negative of the) differential entropy \begin{align} I[c] = \int dx p(x|c) \log(p(x|c)) \end{align} is also maximized at $c_*$?

Note $\int p \log p = (\frac{d}{dm} \int p^m)|_{m=1}$. Or $\int p \log p = \lim_{m\to 1} \log( \int p^m)/(m-1)$. However, the derivative or limit requires knowledge of real $m \geq 1$. Here I am asking does knowledge over all the integers $m$ suffice to constrain the behavior over the reals. I feel it is related to a few concepts: (1) the validity of the replica trick in physics; (2) the moments problem (i.e. whether moments uniquely determine a distribution c.f. Carleman condition, Hausdorff problem etc.).

Here is an example where all entropies share the same maximum: Let $p(x|c) = (1 + 2\sqrt{c(1-c)} \cos(2\pi x)$ for $0 \leq c \leq 1$. Then it is easy to show the maximum of $\int dx p^m(x|c)$ is at $c_* = 1/2$ for any integer $m$. I can see numerically that the maximum of $\int dx p(x|c) \log(x|c)$ is also found at $c_* = 1/2$. This observation can probably be proven by elementary methods in this particular example, but my question is more generally for arbitrary $p$ can we just use knowledge at the integers $m$ to say something about the limiting differential entropy case?

If this is true for the example but not in general, (i) how come it works in this example and (ii) what sufficient conditions on general $p$ can we impose such that we can?

Improve this question
$\endgroup$
7
  • $\begingroup$ In maximum likelihood estimation is used that $\text{argmax} L(\theta;y)=\text{argmax}\log L(\theta)$ by monotonicity of log so maybe the same can be done under the expectation $c*=\text{argmax}I^{(m)}(c)=\text{argmax}\mathbb{E}(p^{m-1}(x|c))=_{?}\text{argmax} \mathbb{E}(\log p^{m-1}(x|c))=\text{argmax}(m-1)\mathbb{E}(\log p(x|c))=\text{argmax}\mathbb{E}(\log p(x|c))=\text{argmax}I(c)$, not sure what is needed to make $\text{argmax} \mathbb{E}(p^{m-1}(x|c))=_{?}\text{argmax} \mathbb{E}(\log p^{m-1}(x|c))$ if even possible although. $\endgroup$
    – Dabed
    Commented Jul 30 at 3:15
  • $\begingroup$ @Dabed thanks! your argument would seem to suggest only the $m=2$ information is sufficient.. $\endgroup$
    – nervxxx
    Commented Jul 30 at 3:29
  • $\begingroup$ Now I think at least one direction should be true as because of Jensen's inequality you should have $$c*=\text{argmax}I^{(m)}(c)=\text{argmax}\mathbb{E}(p^{m-1}(x|c))=\text{argmax}\log\mathbb{E}(p^{m-1}(x|c))\ge_\text{Jensen}\text{argmax} \mathbb{E}(\log p^{m-1}(x|c))=\text{argmax}(m-1)\mathbb{E}(\log p(x|c))=\text{argmax}\mathbb{E}(\log p(x|c))=\text{argmax}I(c)$$, about m=2 not sure I follow why it should be sufficient. $\endgroup$
    – Dabed
    Commented Jul 30 at 4:21
  • $\begingroup$ @Dabed actually, just to understand what you are doing: note the parameter $c = (c_1, c_2, \cdots,c_n)$ can be multi-dimensional (in the particular example it is 1-dim). what does the inequality you showed on $c_*$ mean? Element-wise? $\endgroup$
    – nervxxx
    Commented Jul 30 at 8:06
  • $\begingroup$ Yeah didn't see it treated as multidimensional from the question but doing the unidimensional optimization $c^{*}=\text{argmin} (y-xc)^2=x^{-1}y$ isn't so different from doing least squares $C^{* }=\text{argmin}||Y-XC||^2=(X^tX)^{-1}X^tY$ for example, so in doing $\partial_c\mathbb{E}(p^{m-1}(x|c))=0,\partial_c\mathbb{E}(\log p(x|c))=0$, the $\partial_c$ should be changed to $\nabla_c$, not sure if m equal a specific value should matter for both to give the same value for $c^{*}$. Maybe reformulating the problem in a new question on MO (or MSE too) could give you more and better help. $\endgroup$
    – Dabed
    Commented Jul 30 at 14:07

0

Reset to default

You must log in to answer this question.