If $p(x)$ is a discrete probabilistic density function, one could construct another discrete probabilistic density function proportional to $p(x)[1-p(x)]$ with a corresponding partition function to make the new function sum to $1$. (Rule out the degenerate case where $p(x_0) = 1$ for a unique $x_0$.)

It seems that the new constructed p.d.f has a bigger or equal entropy than $p(x)$. How to prove that? Suggestions are welcomed for any references or direct solution.

  • 4
    $\begingroup$ There is something wrong in the setup: a probability density function may take values more than $1$. And without some continuity assumption on $p$ we could have $p(x) \in \{0,1\}$ for all $x$, so $p(x)(1-p(x)) = 0$ always. $\endgroup$ Jan 12 '19 at 8:58
  • 1
    $\begingroup$ I guess that density is a misleading word here: $p(x) $ is a measure of atom $\{x\} $. $\endgroup$ Jan 12 '19 at 9:35
  • 1
    $\begingroup$ Ah: I see. I somehow misread 'finite support' as 'compact support'. I'm going to edit the question to make it clear that the density is discrete. $\endgroup$ Jan 12 '19 at 9:40

Denote $f(p)=p(1-p)$, $H(p)=-p\log p$, let $p_1,\dots,p_n$ denote all positive probabilities of our distribution, then $\sum p_i=1$, finally denote $s=\sum_i f(p_i)$. Then we need to prove the inequality $$ \sum_i H(f(p_i)/s)\geqslant \sum_i H(p_i). $$ Since $H$ is concave, it suffices to prove that the multiset $\{p_1,\dots,p_n\}$ majorizes the multiset $\{f(p_1)/s,f(p_2)/s,\dots,f(p_n)/s\}$. We may assume that $p_1\geqslant p_2\geqslant \ldots\geqslant p_n$, it implies $f(p_1)\geqslant f(p_2)\geqslant \ldots\geqslant f(p_n)$, since $f(p_i)-f(p_j)=(p_i-p_j)(1-p_i-p_j)$, and $1-p_i-p_j\geqslant 0$ for $i\ne j$. Therefore we need to check for every $k$ that $$\frac{p_1+\dots+p_k}{p_1+\dots+p_n}=p_1+\dots+p_k\geqslant \frac{f(p_1)+\dots+f(p_k)}s,$$ that is equivalent (by multiplying through by the denominators on the far left and far right) to $$(p_1+\dots+p_k)(p_{k+1}^2+\dots+p_n^2)\leqslant (p_1^2+\dots+p_k^2)(p_{k+1}+\dots+p_n),$$ which is a sum of obvious inequalities $p_ip_j^2\leqslant p_i^2 p_j$ over all pairs $i\leqslant k<j$.

  • $\begingroup$ thanks a lot. Is there a theory about such probability transform? $\endgroup$
    – sunxd
    Jan 16 '19 at 22:43
  • $\begingroup$ @sunxd no idea about such theory $\endgroup$ Jan 17 '19 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.