MathOverflow is a question and answer site for professional mathematicians. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Start with a continuous probability distribution given by a density function f(x). Let X be a real random variable whose distribution is given by the probability distribution.

I would like to ask about the following specific way to describe the distribution in terms of the variance of tails.

Given a number t between 0 and 1 let s be such that the probability that $X\ge s$ equals t. (In other words, $\int_s^{\infty}f(x)=t$.) Let $W_X(t)$ be the variance of $X$ conditioned on $X\ge s$.

The function $W_X(t)$ seems an interesting way to describe the original probability distribution, and its tail behavior. I wonder if there is an explicit useful way to move from $W$ back to $f$; Can you give a dictionary between the tail behavior of $f$ and of that of $W$; And an explicit description of $W$ for some "famous" probability distributions. (E.g. the normal distribution, and the Tracy Widom distribution.) And of course if this kind of "transform" is considered in the literature.

UPDATE: Many thanks to Robert and Brendan to their answers. Robert's asymptotic expansion seem amazing. I will be happy to understand the asymptotics as t goes to 0.

Looking at various distribution one sees that when the distribution is exponential, W is constant; when the distribution is normal and thus decays doubly exponentially, W(t) behaves like $\log^{-1}(1/t)$. I am curious how the asymptotic decay of $W(t)$ translates to the tail behavior of the probability distribution. For the tracy Widom distribution when you ask what is the probability that $X \ge M+t \sigma$ you get an expression like $e^{-t^{3/2}}$ and for the probability that $X \le M-t \sigma$ you get something like $e^{-t^3}$. I wonder how these tails behavior can be described in terms of $W_X(t)$ and $W'_X(t)$ respectively, where $W'_X(t)$ deals with conditioning on $X \le s$ rather than on $X \ge s$. So let me form a simple question:

Question: If $W_X(t)$ behaves like $\log^a (1/t)$, for some real $a>0$, what does it say about the tail behavior of $X$. (More generally, how the decay of $W$ translates to the decay of the distribution.)

share|cite|improve this question
This seems to me to be related to something that people in e.g. finance study, called excess-of-loss. I am not sure that the following definition is how it is always done, but my impression is that one studies the expectation of $X$ conditioned on the event $X\geq s$ - $\mathbb{E}[X | X\geq s]$. I am not very familiar with the concept and certainly not with what results that are "out there". However, if one studies the expectation then it does not seem far-fetched that the variance would be of interest as well. – Pierre Jan 17 '12 at 13:13
up vote 6 down vote accepted

For a normal distribution with mean $0$ and standard deviation $\sigma$, I get (with help from Maple) $$W_X(t) = \sigma^2 + \sqrt{\frac{2}{\pi}} \frac{\sigma t}{1 - \text{erf}(t/(\sqrt{2}\sigma))} e^{-t^2/(2 \sigma^2)} + \frac{2 \sigma^2}{\pi (1 - \text{erf}(t/(\sqrt{2}\sigma))^2} e^{-t^2/\sigma^2}$$

As $t \to \infty$, this rather fearsome expression has the asymptotic series

$$W_X(t) \sim {\frac {{{\sigma}}^{4}}{{t}^{2}}}-6\ {\frac {{{\sigma}}^{6}}{{t} ^{4}}}+50\ {\frac {{{\sigma}}^{8}}{{t}^{6}}}-518\ {\frac {{{\sigma}}^{10}}{{t}^{8}}}+6354\ {\frac {{{\sigma}}^{12}}{{t}^{10}}}- 89782\ {\frac {{{\sigma}}^{14}}{{t}^{12}}}+1435330\ {\frac {{{\sigma}}^{16}}{{t}^{14}}}-25625910\ {\frac {{{\sigma}}^{18}}{{t}^{16 }}}+ \ldots$$

where the coefficients appear to be the entries of OEIS sequence A005416 which is described as "Vertex diagrams of order 2n".

share|cite|improve this answer
Thanks a lot, Robert! – Gil Kalai Jan 18 '12 at 17:31

Without knowing an answer to the question, I will note the sub-problem of when $W_X(t)$ is a non-increasing function of $t$. Even though it might appear obvious that truncating a distribution makes the variance smaller, it isn't true in general. However, Mailhot proved it is true if either the distribution function or the density are log-concave, with a similar result for discrete distributions. This applies to lots of common distributions including normal, binomial, Poisson, etc.

E. Mailhot, Une propriété de la variance de certaines lois de probabilité réelles tronquées, C. R. Acad. Sci. Paris Sér. I Math. 301 (1985) 241–244.

share|cite|improve this answer
Thanks a lot, Brendan! – Gil Kalai Jan 18 '12 at 17:31

Expression for the quantile function (the inverse CDF) for the case $E(X) = 0$:

Let $\overline{F}(s) = 1 - F(s)$. Then, $t = \overline{F}(s)$ and

$$\frac{d}{ds} E(X^2|X \geq s) = \frac{d}{ds} \Big( \frac{\int_s^{+\infty}x^2f(x)dx}{\int_s^{+\infty}f(x)dx} \Big) =$$ $$ = \frac{-s^2 f(s) \int_s^{+\infty}f(x)dx+f(s) \int_s^{+\infty}x^2f(x)dx}{\big(\int_s^{+\infty}f(x)dx\big)^2} = $$ $$ = \frac{-s^2 f(s) + f(s) W(\overline{F}(s))}{\overline{F}(s)}$$

On the other hand, $$ \frac{d}{ds} E(X^2|X \geq s) = \frac{d}{ds} W(\overline{F}(s)) = -W'(\overline{F}(s)) f(s)$$ where $W'$ is the derivative of $W$. Therefore, $$ \frac{s^2 - W(\overline{F}(s))}{\overline{F}(s)} = W'(\overline{F}(s))$$ or, using $t$, $$ \frac{(\overline{F}^{-1}(t))^2 - W(t)}{t} = W'(t) $$ and $$ \overline{F}^{-1}(t) = \sqrt{tW'(t) + W(t)}$$ ($\overline{F}^{-1}(t)$ should be positive in some neighborhood of $t = 0$). Then, since $\overline{F}^{-1}(t) = F^{-1}(1-t)$, $$ F^{-1}(x) = \sqrt{(1-x)W'(1-x) + W(1-x)} $$

I hope that it's correct and that it can tell something about the decay of the distribution when $t$ tends to $+0$. Maybe it can also be extended to the case $E(X) \neq 0$.

share|cite|improve this answer
Thanks a lot, Stanislav! – Gil Kalai Jun 11 '12 at 5:57

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.