Assume that we have heavy-tailed distribution $F(x)$ such that \begin{align} F(x)=\mathbb{P}[X\geq x]=x^{-0.5}. \end{align} Then, we produce $N$ independent samples $X_1,X_2,\ldots,X_N$ from this distribution. Assuming that $n(N)\leq N$ is a function of $N$, we pick the $n(N)$ largest amounts among $X_1,X_2,\ldots,X_N$ and produce the summation as \begin{align} S_{n(N)} = \sum_{\ell=1}^{n(N)} X_{i_\ell}, \end{align} Here, $i_1, \ldots, i_{n(N)}$ represent the indices corresponding to the $n(N)$ largest values among the $N$ variables. The question is whether there exists a specific choice of $n(N)$ such that as $N$ approaches infinity, the ratio $n(N)/N$ tends to zero, and we have \begin{align} \lim_{N\rightarrow\infty}\frac{S_{n(N)}}{S_{N}}= 1. \end{align} In other words, does a very small subset of variables represent the whole of them? How can I think about these kinds of problems to solve them?
6
-
$\begingroup$ $F(x)=\mathbb{P}[X\leq x]=x^{-0.5}$ is not here a typo? $\endgroup$– Fedor PetrovCommented Jul 6, 2023 at 10:11
-
$\begingroup$ If it should read as $X\geq x$, then we have a large number of order $n^2$ with high probability, thus the terms which are greater than $\sqrt{n} $ dominate the sum, and we have $o(n) $ such numbers $\endgroup$– Fedor PetrovCommented Jul 6, 2023 at 10:18
-
$\begingroup$ Thank you. You are right, I fixed it. I will be grateful if you write the answer in detail. I would like to know the details of such an analysis. $\endgroup$– Math_YCommented Jul 6, 2023 at 18:04
-
$\begingroup$ I would have written $F(x)= \Pr(X>x) = x^{-0.5} \text{ for } x\ge 1$ (being explicit about that lower bound). And also "a sample of $N$ independent observations from this distribution" rather than "$N$ samples from this distribution." $\qquad$ $\endgroup$– Michael HardyCommented Jul 6, 2023 at 20:32
-
$\begingroup$ Does there exist a distribution that satisfies this? $\endgroup$– usulCommented Jul 7, 2023 at 14:42
|
Show 1 more comment
1 Answer
$\begingroup$
$\endgroup$
10
The probability that all samples are less than $N^{19/10}$ is $(1-N^{-19/20})^{N}$ that tends to 0. The expected number of samples greater than $N^{1/2}$ is $N^{3/4}$, thus, the probability that we have more than $N^{4/5}$ such samples is by Chebyshev inequality at most $N^{-1/20}$,also tends to 0. Therefore, with probability tending to 1 the sum of all but $N^{4/5}$ largest samples does not exceed $N^{3/2}$ while the sum of $N^{4/5}$ largest samples (and even one single largest sample) is at least $N^{19/10}$. It yields that you may take $n(N)=N^{4/5}$. Of course this is not optimal.
answered Jul 7, 2023 at 4:18
-
$\begingroup$ I think the $N^{19/20}$ towards the end is meant to be $N^{19/10}$. $\endgroup$ Commented Jul 7, 2023 at 8:22
-
$\begingroup$ @JamesMartin thank you, fixed $\endgroup$ Commented Jul 7, 2023 at 8:50
-
$\begingroup$ The expected number of samples less than $N^{1/2}$ is $NP(X<N^{1/2})\sim N$. Or, perhaps, I misunderstood what you meant by the expected number of samples less than $N^{1/2}$. $\endgroup$ Commented Jul 7, 2023 at 14:30
-
$\begingroup$ Also, I guess, instead of $N^{19/20}$, you meant $N^{-19/20}$. $\endgroup$ Commented Jul 7, 2023 at 14:31
-
$\begingroup$ @IosifPinelis thank you, fixed this too $\endgroup$ Commented Jul 7, 2023 at 14:34