Let us define a distance metric between two joint probability math functions $p(x,y)$ and $q(x,y)$ as in the following

\begin{align} \sum_{y}\sqrt{\sum_{x}p(x)\left(p(y|x)-q(y|x)\right)^2}. \end{align}

By Jensen inequality, we have

\begin{align*} \sum_{y}\sqrt{\sum_{x}p(x)\left(p(y|x)-q(y|x)\right)^2}&\geq\sum_{x,y}\lvert p(x,y)-q(x,y)\rvert\\ &=\lVert p(x,y)-q(x,y)\rVert_1 \end{align*}

where $\lVert\cdot\rVert_1$ is the total variation distance.

What is the name of this distance? Is there any other relationship between this distance and any other stochastic distances?

  • $\begingroup$ are you assuming $q(x,y) = q(y|x)p(x)$? $\endgroup$ – Memming Sep 11 '17 at 23:43
  • $\begingroup$ some sort of nested Csiszár's f-divergence? $\endgroup$ – Memming Sep 11 '17 at 23:50
  • $\begingroup$ Is this symmetric? $\endgroup$ – Liviu Nicolaescu Sep 12 '17 at 1:42
  • $\begingroup$ It is symmetric for $p(x)=q(x)$. $\endgroup$ – Math_Y Sep 12 '17 at 3:29

\begin{align*} \sum_{y}\sqrt{\sum_{x}p(x)\left(p(y|x)-p(y)\right)^2} &=\sum_{y}p(y)\sqrt{\sum_{x}p(x)\left(\frac{p(y|x)}{p(y)}-1\right)^2}\\ &=\sum_{y}p(y)\sqrt{\sum_{x}p(x)\left(\frac{p(x|y)}{p(x)}-1\right)^2}\\ &=\sum_{y}p(y)\sqrt{\sum_{x}\frac{p^2(x|y)}{p(x)}-2p(x|y)+p(x)}\\ &=\sum_{y}p(y)\sqrt{\sum_{x}\frac{p^2(x|y)}{p(x)}-1}\\ &=\mathbb{E}_Y\sqrt{\chi^2\left(p(x|y)\parallel p(x)\right)} \end{align*} where $\chi^2(\cdot||\cdot)$ is chi-square distance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.