Given normal distributions with a single positional and variation parameter each, $p_1=\big[\mu_1, \sigma_1\big]$, $p_2=\big[\mu_2, \sigma_2\big]$, we define their Jensen-Shannon divergence as: $$ \text{JS}(p_1, p_2)=\big|\int_{\mathbb{R}}\frac{p_1+p_2}{2}\ln\big(\frac{p_1+p_2}{2}\big)-\frac{1}{2}\big(p_1\ln p_1+p_2\ln p_2\big)dx\big| $$ We write $p_1(s)=\big[\mu_1,\sigma_1/\sqrt{s}\big]$, $p_2(s)=\big[\mu_2,\sigma_2/\sqrt{s}\big]$ to signify their corresponding sampling distributions, $s>0$.

Question: Are there any resources that establish a relationship, ideally some identity, between the $\text{JS}(p_1, p_2)$ and $\text{JS}(p_1(s), p_2(s))$?

EDIT For simplicity, $1/\sqrt{s}=\lambda$, and we will refer to the distributions above in terms of their coordinates (as brackets). We will superscript the Jensen-Shannon by $\lambda$ to say that each distribution is denormalized by a factor $\lambda$. Since the Jensen-Shannon is $1$-homogenous ($f(\lambda x)=\lambda f(x)$), we can extract $\lambda$ as below, then translating by $x\mapsto x+\mu_1$, $$ \frac{1}{\lambda}\text{JS}^{\lambda}\big([0, \sigma_1\lambda],[\mu_2-\mu_1, \sigma_2\lambda]\big) $$ We scale by $x\mapsto x\lambda$ to eliminate the $\lambda$ factors from the $\sigma$ terms. $$ =\text{JS}\big([0, 1], [(\mu_2-\mu_1)/\lambda, \sigma_2/\sigma_1]) $$ Of course for the original divergence, similar operations give us $$ \text{JS}(p_1, p_2)=\text{JS}\big([0, 1], [(\mu_2-\mu_1), \sigma_2/\sigma_1]\big) $$ So really the question becomes comparing the Jensen-Shannon divergences of two distributions, the first $[0, 1]$ and the second $[\mu_2-\mu_1, \sigma_2/\sigma_1]$, where the second Jensen-Shannon has scaled the positional argument by $1/\lambda$.

Therefore I propose we consider this question to be a multiplicative property (if any) of the position whereas the translation-invariance property is an additive property.

  • $\begingroup$ Do you mean the corresponding sampling distribution of the means as a function of sample size $s$? $\endgroup$
    – Memming
    Apr 18, 2019 at 14:22
  • $\begingroup$ Yes, I am wondering if there is some inequality or relationship between the Jensen-Shannon of distributions and their respective $s$ sampling distributions. I understand the Jensen-Shannon is invariant under translation of position ($\mu_1+\lambda, \mu_2+\lambda$), so I'm curious to see if there similar properties w.r.t. the variances. $\endgroup$ Apr 18, 2019 at 16:04


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy