12
$\begingroup$

$X\in \mathbb{R}$. Which distribution $P(X)$ has the highest possible entropy given its expected value, variance, skewness, and kurtosis? Is it an exponential family distribution of the form $P(X) \propto \exp(a\cdot x +b\cdot x^2 + c\cdot x^3 +d\cdot x^4)$ in analogy to the normal distribution being the maximum-entropy distribution given mean and variance? If so, where can I read about it?

I am interested in this question because I want to model data with high skewness and high kurtosis, but I am still looking for an appropriate distribution.

$\endgroup$
2
  • $\begingroup$ Perhaps better suited for stats.SE $\endgroup$
    – Memming
    Sep 10, 2013 at 4:17
  • 9
    $\begingroup$ Probability theory is certainly on-topic here. $\endgroup$ Sep 10, 2013 at 5:06

1 Answer 1

10
$\begingroup$

Yes, your $P(X) \propto \exp(a\cdot x +b\cdot x^2 + c\cdot x^3 +d\cdot x^4)$ maximises the entropy $-\int P(X){\rm log} P(X)dX$ for prescribed first four moments, if the skewness and curtosis lie in a certain range:

M. Rockinger and E. Jondau, Entropy densities with an application to autoregressive conditional skewness and kurtosis (2002).

This simple generalization of the normal distribution holds also if higher moments are prescribed, provided that the highest prescribed moment is even. If, for example, only mean, variance, and skewness, are prescribed, then $d$ would be zero and the distribution fails to normalize. In this case the maximum entropy distribution exists, but it has a more complicated form, as discussed here.

You'll still have to determine the coefficients $a,b,c,d$ from the given first four moments. The cited 2002 paper gives a description of an efficient method. There may be no solution (typically, if the skewness is too large relative to the kurtosis), meaning that a maximum entropy solution of this simple form does not exist (see Figure 1 of the paper). The combination of a prescribed skewness and kurtosis typically leads to a bimodal distribution (see Figures 2 and 3), which may or may not be desirable for your application.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.