Given a parametric family of distributions $\{p_\theta|\theta \in \Theta\}$, one can show that under some regularity conditions, the following approximation is valid $$\operatorname{KL}(p_\theta\| p_{\theta + d \theta}) = d \theta^TF(\theta)d\theta + \mathcal O(\|d\theta\|^3), $$ where $$F(\theta)_{ij} := \mathbb E_{x \sim p_\theta}\left[\frac{\partial^2}{\partial \theta_i\partial \theta_j}\log(p_\theta(x))\right] $$ is the Fisher information matrix of $p_\theta$. A very rough sketch of the proof can be found on [wikipedia][1]. Question 1 ========== Is there such an approximation formula for the [Wassertein distance][2] or other measures of discrepancy between probability distributions ? Question 2 ========== Same question, specialized to [$f$-divergences][3] (of which KL is a particular case). [1]: https://en.wikipedia.org/wiki/Fisher_information#Jeffreys_prior_in_Bayesian_statistics [2]: https://en.wikipedia.org/wiki/Wasserstein_metric [3]: https://en.wikipedia.org/wiki/F-divergence