5
$\begingroup$

Given a conditional distribution $P_{Y\mid X}$ I'd like to find the prior distribution $P_X$ that maximizes the mutual information $I(X;Y)$ with $P_Y(y)=\int P_{Y\mid X}(y\mid x)P_X(x) \, \text{d}x$ (this corresponds to finding the channel capacity $C(X;Y):=\max_{P_X}I(X;Y)$) subject to the constraint $E_{P_X}[-\log(X)]=a$.

In my particular case, $P_{Y\mid X}$ is a Bernoulli distribution with $X$ as its parameter and I'm looking for a distribution over the parameter.

My intuition would tell me this should be some Beta-distribution (something like $\text{Beta}(1/a,1)$?!) in my particular case, but I don't know how to approach such a problem, much less in the general case.

Could anyone point me in the right direction?

Improve this question
$\endgroup$
6
  • $\begingroup$ What (and why) is the constraint $E[-\log(X)]=a$? $\endgroup$
    – Ashok
    Feb 27 '13 at 13:09
  • $\begingroup$ Sorry, $E[-\log(X)]$ means the expected value of the random variable $X':=-\log(X)$. It's a little bit complicated to explain where this constraint comes from. In my particular case it's because the above stuff is a discrete approximization of a poisson process, where "heads" are thrown at a varying rate R. If time is discretized into little bins, this gives, for each of those bins, the probability $X$ to have no "heads" in that bin where $X = \exp(-R)$. The expected value of the rate $E[R]=E[-\log(X)]=a$ is fixed. If the constraint is the problem, $E[X]=a$ would also be ok. $\endgroup$
    – user31757
    Feb 27 '13 at 16:21
  • $\begingroup$ What is the range of $X$ (because you are saying $P_{Y|X}$ is Bernoulli with parameter $X$)? $\endgroup$
    – Ashok
    Feb 28 '13 at 12:08
  • $\begingroup$ Sorry, I perhaps was a bit too unspecific there. $P_{Y|X}{Y=1|X=x}=x$ and $P_{Y|X}(Y=0|X=x)=(1-X)$ where $Y\in\{0,1\}$ and $X\in[0;1]$. $\endgroup$
    – user31757
    Mar 2 '13 at 3:48
  • $\begingroup$ wupps, should've been $P_{Y|X}(Y=1|X=x)=x$., ofc. $\endgroup$
    – user31757
    Mar 2 '13 at 3:50
1
$\begingroup$

Solution to the special case of $a=0.5$ under the less restrictive case $E[X]=a$. This case is trivial and is solved by $P(X=0)=P(X=1)=0.5$ in which the channel capacity is one which is the maximal value for your channel.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.