A Variance-Tail Description for Continuous Probability Distributions

Question

Start with a continuous probability distribution given by a density function f(x). Let X be a real random variable whose distribution is given by the probability distribution.

I would like to ask about the following specific way to describe the distribution in terms of the variance of tails.

Given a number t between 0 and 1 let s be such that the probability that $X\ge s$ equals t. (In other words, $\int_s^{\infty}f(x)=t$.) Let $W_X(t)$ be the variance of $X$ conditioned on $X\ge s$.

The function $W_X(t)$ seems an interesting way to describe the original probability distribution, and its tail behavior. I wonder if there is an explicit useful way to move from $W$ back to $f$; Can you give a dictionary between the tail behavior of $f$ and of that of $W$; And an explicit description of $W$ for some "famous" probability distributions. (E.g. the normal distribution, and the Tracy Widom distribution.) And of course if this kind of "transform" is considered in the literature.

UPDATE: Many thanks to Robert and Brendan to their answers. Robert's asymptotic expansion seem amazing. I will be happy to understand the asymptotics as t goes to 0.

Looking at various distribution one sees that when the distribution is exponential, W is constant; when the distribution is normal and thus decays doubly exponentially, W(t) behaves like $\log^{-1}(1/t)$. I am curious how the asymptotic decay of $W(t)$ translates to the tail behavior of the probability distribution. For the tracy Widom distribution when you ask what is the probability that $X \ge M+t \sigma$ you get an expression like $e^{-t^{3/2}}$ and for the probability that $X \le M-t \sigma$ you get something like $e^{-t^3}$. I wonder how these tails behavior can be described in terms of $W_X(t)$ and $W'_X(t)$ respectively, where $W'_X(t)$ deals with conditioning on $X \le s$ rather than on $X \ge s$. So let me form a simple question:

Question: If $W_X(t)$ behaves like $\log^a (1/t)$, for some real $a>0$, what does it say about the tail behavior of $X$. (More generally, how the decay of $W$ translates to the decay of the distribution.)

Answer 1

For a normal distribution with mean $0$ and standard deviation $\sigma$, I get (with help from Maple) $$W_X(t) = \sigma^2 + \sqrt{\frac{2}{\pi}} \frac{\sigma t}{1 - \text{erf}(t/(\sqrt{2}\sigma))} e^{-t^2/(2 \sigma^2)} + \frac{2 \sigma^2}{\pi (1 - \text{erf}(t/(\sqrt{2}\sigma))^2} e^{-t^2/\sigma^2}$$

As $t \to \infty$, this rather fearsome expression has the asymptotic series

$$W_X(t) \sim {\frac {{{\sigma}}^{4}}{{t}^{2}}}-6\ {\frac {{{\sigma}}^{6}}{{t} ^{4}}}+50\ {\frac {{{\sigma}}^{8}}{{t}^{6}}}-518\ {\frac {{{\sigma}}^{10}}{{t}^{8}}}+6354\ {\frac {{{\sigma}}^{12}}{{t}^{10}}}- 89782\ {\frac {{{\sigma}}^{14}}{{t}^{12}}}+1435330\ {\frac {{{\sigma}}^{16}}{{t}^{14}}}-25625910\ {\frac {{{\sigma}}^{18}}{{t}^{16 }}}+ \ldots$$

where the coefficients appear to be the entries of OEIS sequence A005416 https://oeis.org/A005416 which is described as "Vertex diagrams of order 2n".

Answer 2

Without knowing an answer to the question, I will note the sub-problem of when $W_X(t)$ is a non-increasing function of $t$. Even though it might appear obvious that truncating a distribution makes the variance smaller, it isn't true in general. However, Mailhot proved it is true if either the distribution function or the density are log-concave, with a similar result for discrete distributions. This applies to lots of common distributions including normal, binomial, Poisson, etc.

E. Mailhot, Une propriété de la variance de certaines lois de probabilité réelles tronquées, C. R. Acad. Sci. Paris Sér. I Math. 301 (1985) 241–244.

Answer 3

Expression for the quantile function (the inverse CDF) for the case $E(X) = 0$:

Let $\overline{F}(s) = 1 - F(s)$. Then, $t = \overline{F}(s)$ and

$$\frac{d}{ds} E(X^2|X \geq s) = \frac{d}{ds} \Big( \frac{\int_s^{+\infty}x^2f(x)dx}{\int_s^{+\infty}f(x)dx} \Big) =$$ $$ = \frac{-s^2 f(s) \int_s^{+\infty}f(x)dx+f(s) \int_s^{+\infty}x^2f(x)dx}{\big(\int_s^{+\infty}f(x)dx\big)^2} = $$ $$ = \frac{-s^2 f(s) + f(s) W(\overline{F}(s))}{\overline{F}(s)}$$

On the other hand, $$ \frac{d}{ds} E(X^2|X \geq s) = \frac{d}{ds} W(\overline{F}(s)) = -W'(\overline{F}(s)) f(s)$$ where $W'$ is the derivative of $W$. Therefore, $$ \frac{s^2 - W(\overline{F}(s))}{\overline{F}(s)} = W'(\overline{F}(s))$$ or, using $t$, $$ \frac{(\overline{F}^{-1}(t))^2 - W(t)}{t} = W'(t) $$ and $$ \overline{F}^{-1}(t) = \sqrt{tW'(t) + W(t)}$$ ($\overline{F}^{-1}(t)$ should be positive in some neighborhood of $t = 0$). Then, since $\overline{F}^{-1}(t) = F^{-1}(1-t)$, $$ F^{-1}(x) = \sqrt{(1-x)W'(1-x) + W(1-x)} $$

I hope that it's correct and that it can tell something about the decay of the distribution when $t$ tends to $+0$. Maybe it can also be extended to the case $E(X) \neq 0$.