2
$\begingroup$

Let $0<a<b<c$ be distinct positive reals.

Define four different probability distributions:

$$\mathcal{P}_{ab}:P_{a,ab}=\frac{a}{a+b}=1-P_{b,ab}$$ $$\mathcal{P}_{bc}:P_{b,bc}=\frac{b}{b+c}=1-P_{c,bc}$$ $$\mathcal{P}_{ca}:P_{c,ca}=\frac{c}{c+a}=1-P_{a,ca}$$ $$\mathcal{P}_{abc}:P_{a,abc}=\frac{a}{a+b+c},\mbox{ }P_{b,abc}=\frac{b}{a+b+c},\mbox{ }P_{c,abc}=\frac{c}{a+b+c}$$

Does Shannon entropy of a random variable from distribution $\mathcal{P}_{abc}$ dominate the other three for all $a,b,c\in\Bbb R^+$ such that $0<a<b<c$?

In general when does $\mathcal{P}_{abc}$ dominate?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Does Shannon entropy of a random variable from distribution $\mathcal{P}_{abc}$ dominate the other three for all $a,b,c\in\Bbb R^+$ such that $0<a<b<c$?

No.

On a sample space $\{A,B,C\}$ we can think of $\mathcal P_{ab}$ as just $\mathcal P_{abc}$ conditioned on the event that $C$ did not happen.

If the probability $x$ of $C$ is very large and the probabilities of $A$ and $B$ are equal and small, then the entropy of $\mathcal P_{ab}$ will be larger than that of $\mathcal P_{abc}$ as you can verify by looking at the graph of $$-2((1-x)/2)\log_2((1-x)/2)-x\log_2(x),\quad x\in [1/3, 1] $$

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for the wonderful insight. So this is my question actually from which the above spun out. Say you have positive $\{a_i\}_{i=1}^n$ and you have $p_i=\frac{a_i}{\sum_{i=1}^na_i}$, then assume you have a $C$ such that $C<2a_n\ll\sum_{i=1}^na_i$ (that is $C$ is not very large), then define $q_i=\frac{a_i}{C+\sum_{i=1}^na_i}$ and $q_{n+1}=\frac{C}{C+\sum_{i=1}^na_i}$. Now does entropy of $q$ dominate entropy of $p$? $\endgroup$
    – Turbo
    Commented Dec 24, 2014 at 1:52
  • 1
    $\begingroup$ Interesting. Maybe a separate question about that? $\endgroup$
    – Bjørn Kjos-Hanssen
    Commented Dec 24, 2014 at 2:23
  • $\begingroup$ Added mathoverflow.net/questions/191397/…. $\endgroup$
    – Turbo
    Commented Dec 24, 2014 at 4:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .