Consider the following optimization problem: \begin{equation} \max_{\mu \in \mathcal{M}} \int \log\left( \int e^{\alpha U(x,y)} d\mu(y) \right) d\nu(x) \end{equation} where $\mathcal{M}$ is the space of probability measures on the real line endowed with the Borel $\sigma$-algebra, and $\nu \in \mathcal{M}$ is given. $U$ is a $\mathcal{C}^\infty$ function, is strictly concave in $y$, and $y \mapsto U(x,y)$ is uniquely maximized at $y=x$.

Here is the question: Is it possible to characterize the solution of this optimization problem in terms of $\nu$ and $U$, and in particular when $\alpha \to \infty$? If so, how?


(a) Intuitively, $\mu$ should put most mass where $\nu$ does and where $U(x,x)$ is largest. However, I was unable to show a result that makes this statement precise.

(b) I was able to show that, if $U$ is analytic in both its arguments, the support of $\mu$ is either (i) discrete with no accumulation point other than infinity, either (ii) the support of $\mu$ is the same as the support of $\nu$. In particular, if $\nu$ has bounded support, $\mu$ will generically be supported on a finite set of points.

(c) One can see analogies with Kullback-Leibler divergence minimization/entropy maximization, but I am relatively new to these questions and I was unable to relate both problems in a precise way. One related although a priori different problem is the one here: Optimization over space of probability measures

(d) Feel free to impose additional structure on the problem. In particular, one can start with the case in which $\nu$ is absolutely continuous with respect to the the Lebesgue measure. However, note that the main difficulty with the problem at hand is that $\mu$ may well not be absolutely continuous, which occurs in many cases.

I am new to information theory and related divergence minimization problems, so maybe this question has an easy answer. In any case any input will be greatly appreciated! Thank you very much in advance!

  • $\begingroup$ Have you tried to see if this can be regarded as having a q-analogue that is a Wasserstein distance? More generally, I would suspect that transportation theory might inform a question such as yours. $\endgroup$ – Steve Huntsman Jun 21 '16 at 12:29
  • $\begingroup$ @SteveHuntsman: I had not thought of that, I'll look into it. Thanks! As a general comment to the question: any method of solution that derives an approximation to the optimal probability measure as alpha goes to infinity is very much welcome. $\endgroup$ – Adrien Jun 21 '16 at 13:02
  • $\begingroup$ Did you try considering a discrete problem — measure $\mu$ supported on a given finite set $x_1,\dots,x_n$ (thus parametrized by the weights of atoms $p_1,\dots,p_n$)? Then it becomes a standard calculus optimization problem (with a Lagrange multiplier due to the condition $p_1+\dots+p_n=1$ and conditions $p_i\ge 0$). And then you may try passing to the limit (for instance, in the optimality condition) as the set of points tends to be dense... $\endgroup$ – Victor Kleptsyn Jun 23 '16 at 10:28
  • $\begingroup$ @VictorKleptsyn: Yes I have tried that, and the approach would also work as soon as $\mu$ has a density. It allows to derive first-order conditions that indeed characterize the pdf/pmf. One potential approach is then to use Laplace method/Varadhran's lemma -like arguments to expand the resulting integrals when $\alpha \to \infty$ and characterize $\mu$. However this poses 3 difficulties that I could not resolve. $\endgroup$ – Adrien Jun 23 '16 at 12:52
  • $\begingroup$ @VictorKleptsyn [continued]: the difficulties being: : (i) a priori $\mu$ is not guaranteed to have a density if it is continuous, (ii) one needs at least to show a priori that $(\alpha,\mu_\alpha)$ satisfies at least large deviation principle, and (iii) the expansion approach also poses normalization issues (but these could probably be overcome). $\endgroup$ – Adrien Jun 23 '16 at 12:55

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.