8
$\begingroup$

Is there an extension of maximum entropy probability distributions for function spaces?

For $\mathbb{R}^n$ and discrete spaces, there is much literature about this problem under names such as "non-informative priors", "maximum entropy distributions", "Jeffrey's priors", and the like.

There is an extension to locally compact topological groups, where the Haar measure $U$ takes the place of the Lebesgue measure, and one looks for measures $P$ minimizing the information divergence, $$D(P||U):= \begin{cases} \int log \frac{dP}{dU} dP, & \text{ if } P\ll U; \\\ \infty, & \text{else.}\end{cases}$$

However, I've found little about this in the infinite dimensional setting. Can the concept of maximum entropy priors be generalized to (some class of) function spaces, or is the idea of entropy fundamentally incompatible with spaces that are not locally compact?

Notes,

$\endgroup$
5
$\begingroup$

There is work on infinite-dimensional exponential families of measures which might be what you are looking for.

There are these possible references:

The first paper provides such distributions on Sobolev spaces. Also, the references in that paper are helpful about using infinite-dimensional entropy optimizing-based measures in order to construct priors for use in Bayesian non-parametric statistics.

Edit: I added an additional reference that is also relevant to the topic.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks, this was very helpful. I've looked around a lot over the last couple weeks and my conclusion is that exponential families are the most natural generalization of maximum entropy distributions to infinite dimensional spaces. To future readers, it is instructive to consider the finite dimensional case with absolutely continuous distributions, where (under mild conditions) one can explicitly show that exponential families extremize the constrained convex optimization problem $max_f H(f)$ such that $\mathbb{E}_f(g_i)=t_i$. $\endgroup$ – Nick Alger Nov 1 '12 at 5:56
  • $\begingroup$ Hi, can you go more into detail about what references are about infinite-dimensional entropy optimizing-based measures? I don't see any references to the priors being constructed as entropy maximizing. $\endgroup$ – bringingdownthegauss May 30 '19 at 19:12
3
$\begingroup$

Cover and Thomas's Elements of Information Theory has a chapter on maximum entropy stochastic processes. The relevant quantity in that case is the entropy rate. See section 12.5, for example, which is visible in Google books.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.