Skip to main content
2 of 2
added 69 characters in body

Smallest $\mathrm{D}(Q\|P)$ given fixed marginals $\mathrm{D}(Q_X\|P_X)$ and $\mathrm{D}(Q_Y\|P_Y)$

Let $P$ be a distribution on a set $U\times V$ with marginal distributions $P_X$ and $P_Y$.

Suppose we have two values $d_x, d_y\in\mathbb R$, and we want to find the distribution $Q$ absolutely continuous with respect to $P$ that minimizes

$$\mathrm{D}(Q\|P)$$

given $\mathrm{D}(Q_X\|P_X) = d_x$ and $\mathrm{D}(Q_Y\|P_Y) = d_y$.

What can we say about $Q$? Do we even know that such a distribution exists? Or can there be an infinite sequence of $Q'$ with respectively smaller $\mathrm{D}(Q'\|P)$? I'm particularly interested in what the marginals of $Q$ look like.

If instead we had fixed $E_P[(X,Y)]$ (assuming now $X,Y\in\mathbb R$), we know from e.g. Corollary 12.1. in Yury Polyanskiy and Yihong Wu that the distribution minimizing $\mathrm{D}(Q\|P)$ given $E_Q[(X,Y)]=E_P[(X,Y)]$ is a titled version of $P$. That is $dQ = \frac{e^{\phi}}{Ee^{\phi}}dP$ where $\phi = \lambda_1 X+\lambda_2 Y$.

I wonder if there is a similar simple family of distributions one might restrict to given my requirement on $Q$?

Below is a visualization belonging to said Corollary 12.1.

enter image description here