Return to Answer

Basically speaking, the theory of asymptotics using the framework of decision theory is much tougher than most mathematicians think nowadays if they ever read into [Le Cam]'s exposition. This kind of difficulty mostly arise from two sources, from my perspective. The first one is the typical difficulty in determination of loss function, convexity is convenient yet concavity is a more faithful description of boundary behavior (when sequence of rules tends to $\infty$, in most cases we consider the rules indexed by sample sizes $n$.), this flaws are very carefully(probably over-carefully discussed in pp.16-23 of [Le Cam]). The second one is closer to what you think in OP, that is the inconsistency between local and global asymptotic behaviors, which is also discussed in Chap.10-11 of [Le Cam]. This problem is becoming more and more central concern in recent research, you may also want to read my post Is there a result that provides the bootstrap is valid if and only if the statistic is smooth?, which explained why sometimes seemingly unnecessary "linearity conditions" will be imposed on some asymptotic results.

There are quite a few examples that concentration bound disagrees with asymptotic bound as I mentioned above shown in [Talagrand] and [Le Cam]Chap 10-11. But the most famous example that fits your description should be the order statistics from extreme value theory, its asymptotic distribution is exponential (Limiting distribution of the first order statistic of a general distribution) while the distribution of finite sample order statistics $(X_{(1)},\cdots,X_{(n)})$ always depends on the underlying distribution itself. Not surprisingly the Efron-Stein concentration bound is much looser in this case. So what I said is that such disagreement between concentration bound and asymptotic bound is rather common.

Basically speaking, the theory of asymptotics using the framework of decision theory is much tougher than most mathematicians think nowadays if they ever read into [Le Cam]'s exposition. This kind of difficulty mostly arise from two sources, from my perspective. The first one is the typical difficulty in determination of loss function, convexity is convenient yet concavity is a more faithful description of boundary behavior (when sequence of rules tends to $\infty$, in most cases we consider the rules indexed by sample sizes $n$.), this flaws are very carefully(probably over-carefully discussed in pp.16-23 of [Le Cam]). The second one is closer to what you think in OP, that is the inconsistency between local and global asymptotic behaviors, which is also discussed in Chap.10-11 of [Le Cam]. This problem is becoming more and more central concern in recent research, you may also want to read my post Is there a result that provides the bootstrap is valid if and only if the statistic is smooth?, which explained why sometimes seemingly unnecessary "linearity conditions" will be imposed on some asymptotic results.

There are quite a few examples that concentration bound disagrees with asymptotic bound as I mentioned above shown in [Talagrand] and [Le Cam]Chap 10-11. But the most famous example that fits your description should be the order statistics from extreme value theory, its asymptotic distribution is exponential (Limiting distribution of the first order statistic of a general distribution) while the distribution of finite sample order statistics $(X_{(1)},\cdots,X_{(n)})$ always depends on the underlying distribution itself. Not surprisingly the Efron-Stein concentration bound is much looser in this case. So what I said is that such disagreement between concentration bound and asymptotic bound is rather common.

Although I do not quite agree with your opinion, this question is actually quite interesting, therefore I want to focus on a more specific version of the problem you raise, and then return to your question that "What are some other surprising results of finite sample statistics?" The question you raise concerns

Thank you for sharing your thoughts!

The other is actually the demanding need of studying the robustness of certain estimators(M-estimators). When an accurate approximation is available, we do not have to worry about multiple Hadamard differentiation when studying the influence of certain observations. Huber and Rochetti even updated this contribution into Huber's epic work [Huber] when revised.

What [Devroye] showed is a concentration bound (a bound that holds with hihg probability). This is not uncommon in mathematics, say subharmonic functions sequence do not always converge to subharmonic functions, how can we expect that a subharmonic estimator(empirical mean) sequence converges to a harmonic limit(Gaussian when taking $n\rightarrow \infty$)? More obviously for finite sample you always have union bound, how about that asymptotically?

There are quite a few examples that concentration bound disagrees with asymptotic bound as I mentioned above shown in [Talagrand] and [Le Cam]Chap 10-11. But the most famous example that fits your description should be the order statistics from extreme value theory, its asymptotic distribution is exponential (Limiting distribution of the first order statistic of a general distribution) while the distribution of finite sample order statistics $(X_(1),\cdots,X_(n))$ always depends on the underlying distribution itself. Not surprisingly the Efron-Stein concentration bound is much looser in this case. So what I said is that such disagreement between concentration bound and asymptotic bound is rather common.

Although I do not quite agree with your opinion, this question is actually quite interesting, therefore I want to focus on a more specific version of the problem you raise, and then return to your question that "What are some other surprising results of finite sample statistics?" We firstly considered the small sample approximation, and then a correction and discussion about your observation: the different between small/finite sample behavior and large/asymptotic behavior. Thank you for sharing your thoughts!

The other is actually the demanding need of studying the robustness of certain estimators(M-estimators). When an accurate approximation is available, we do not have to worry about multiple Hadamard differentiation when studying the influence of certain observations. Huber and Ronchetti even updated this contribution into Huber's epic work [Huber] when revised.

What [Devroye] showed is a concentration bound (a bound that holds with high probability). This is not uncommon in mathematics, say subharmonic functions sequence do not always converge to subharmonic functions, how can we expect that a subharmonic estimator(empirical mean) sequence converges to a harmonic limit(Gaussian when taking $n\rightarrow \infty$)? More obviously for finite sample you always have union bound, how about that asymptotically?

There are quite a few examples that concentration bound disagrees with asymptotic bound as I mentioned above shown in [Talagrand] and [Le Cam]Chap 10-11. But the most famous example that fits your description should be the order statistics from extreme value theory, its asymptotic distribution is exponential (Limiting distribution of the first order statistic of a general distribution) while the distribution of finite sample order statistics $(X_{(1)},\cdots,X_{(n)})$ always depends on the underlying distribution itself. Not surprisingly the Efron-Stein concentration bound is much looser in this case. So what I said is that such disagreement between concentration bound and asymptotic bound is rather common.

For the question you asked, there is actually a field of research that is called "small sample asymptotics" [Field&Ronchetti]. This field has at least two motivations, one is mentioned in [Field&Ronchetti] Preface:

There are quite a few examples that concentration bound disagrees with asymptotic bound as I mentioned above shown in [Talagrand] and [Le Cam]Chap 10-11. But the most famous example that fits your description should be the order statistics from extreme value theory, its asymptotic distribution is exponential Limiting distribution of the first order statistic of a general distributionwhile the distribution of finite sample order statistics $(X_(1),\cdots,X_(n))$ always depends on the underlying distribution itself. Not surprisingly the Efron-Stein concentration bound is much looser in this case. So what I said is that such disagreement between concentration bound and asymptotic bound is rather common.

If you are concerned with the asymptotic behavior of finite sample, a relevant field of research that is called "small sample asymptotics" [Field&Ronchetti]. This field has at least two motivations, one is mentioned in [Field&Ronchetti] Preface:

There are quite a few examples that concentration bound disagrees with asymptotic bound as I mentioned above shown in [Talagrand] and [Le Cam]Chap 10-11. But the most famous example that fits your description should be the order statistics from extreme value theory, its asymptotic distribution is exponential (Limiting distribution of the first order statistic of a general distribution) while the distribution of finite sample order statistics $(X_(1),\cdots,X_(n))$ always depends on the underlying distribution itself. Not surprisingly the Efron-Stein concentration bound is much looser in this case. So what I said is that such disagreement between concentration bound and asymptotic bound is rather common.