MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Given a conditional distribution $P_{Y|X}$ I'd like to find the prior distribution $P_X$ that maximizes the mutual information $I(X;Y)$ with $P_Y(y)=\int P_{Y|X}(y|x)P_X(x)\text{dx}$ (this corresponds to finding the channel capacity $C(X;Y):=\max_{P_X}I(X;Y)$) subject to the constraint $E_{P_X}[-\log(X)]=a$.

In my particular case, $P_{Y|X}$ is a Bernoulli distribution with $X$ as its parameter and I'm looking for a distribution over the parameter.

My intuition would tell me this should be some Beta-distribution (something like $\text{Beta}(1/a,1)$?!) in my particular case, but I don't know how to approach such a problem, much less in the general case.

Could anyone point me in the right direction?

share|cite|improve this question
What (and why) is the constraint $E[-\log(X)]=a$? – Ashok Feb 27 '13 at 13:09
Sorry, $E[-\log(X)]$ means the expected value of the random variable $X':=-\log(X)$. It's a little bit complicated to explain where this constraint comes from. In my particular case it's because the above stuff is a discrete approximization of a poisson process, where "heads" are thrown at a varying rate R. If time is discretized into little bins, this gives, for each of those bins, the probability $X$ to have no "heads" in that bin where $X = \exp(-R)$. The expected value of the rate $E[R]=E[-\log(X)]=a$ is fixed. If the constraint is the problem, $E[X]=a$ would also be ok. – user31757 Feb 27 '13 at 16:21
What is the range of $X$ (because you are saying $P_{Y|X}$ is Bernoulli with parameter $X$)? – Ashok Feb 28 '13 at 12:08
Sorry, I perhaps was a bit too unspecific there. $P_{Y|X}{Y=1|X=x}=x$ and $P_{Y|X}(Y=0|X=x)=(1-X)$ where $Y\in\{0,1\}$ and $X\in[0;1]$. – user31757 Mar 2 '13 at 3:48
wupps, should've been $P_{Y|X}(Y=1|X=x)=x$., ofc. – user31757 Mar 2 '13 at 3:50

Solution to the special case of $a=0.5$ under the less restrictive case $E[X]=a$. This case is trivial and is solved by $P(X=0)=P(X=1)=0.5$ in which the channel capacity is one which is the maximal value for your channel.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.