Timeline for Optimization of relative entropy
Current License: CC BY-SA 2.5
4 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 26, 2010 at 22:52 | comment | added | skypemesm | Yes. I agree with you. There has to be an upper bound on $BS_i$. However, relative entropy is only defined if P and Q both sum to 1 and if Q(i) > 0 for any i such that P(i) > 0. If the quantity 0log0 appears in the formula, it is interpreted as zero [source:wikipedia]. And $\lambda$ is the lagrange multiplier here. Thanks for replying. | |
| Oct 25, 2010 at 21:05 | comment | added | Suvrit | Note that if p(S)=0 for any index S, then taking q(S) > 0 maximizes your relative entropy. So some more care is needed in formulating your problem. Also, to take care of constraint, use the idea of "Lagrange multipliers". From your formulation above, it seems that you have written some sort of Lagrangian, and then merely taken derivatives. The way it is written above, letting BS_i increase without bound also maximizes $\Lambda$. Can you edit your problem to formally state your optimization task? | |
| Oct 25, 2010 at 20:44 | comment | added | skypemesm | Yes I guess we can minimize the mutual information or maximize relative entropy, so that the output distribution is as different as possible from the input distribution. But with mutual information, joint probability and marginal probabilities come in, and it becomes difficult. I agree with the implicit constraint that q has to be a distribution i.e. sum up to 1. How do I proceed from here then? | |
| Oct 25, 2010 at 20:23 | history | answered | Suvrit | CC BY-SA 2.5 |