Suppose an urn contains unknown but non-random numbers of red and green marbles, and I take a random sample of a known and non-random size. Observing the numbers of red and green marbles in the sample, I need to hazard my best guess as to the proportion (not the total number) of red marbles in the urn.

The number of distinct marbles I observe is likely to be larger if the sample is taken without replacement than with replacement. Therefore, there should be less uncertainty in my estimate if I sample without replacement.

THEREFORE the standard deviation of the random number of red marbles in my sample is smaller if I sample without replacement than with.

Clearly the above is not a logically rigorous argument. (Of course, one can establish the same result by easy standard arguments, but that will be off-topic here.)

Now suppose I take a sample of $n$ independent measurements $X_1,\dots,X_n$ from a population that is normally distributed with unknown mean $\mu$ and with variance $\sigma^2$. Let $\overline{X} = (X_1+\cdots+X_n)/n$ and $S^2 = \sum_{i=1}^n (X_i - \overline{X})^2/(n-1)$ be respectively the sample mean and sample variance. Then $$ \frac{\overline{X} - \mu}{\sigma/\sqrt{n}} $$ is normally distributed with known mean and variance 0 and 1 respectivly, so the endpoints of a confidence interval for $\mu$ are $$ \overline{X} \pm A \frac{\sigma}{\sqrt{n}} $$ where $A$ is a suitable percentage point of the normal distribution (provided (unrealistically) that $\sigma$ is known).

And $$ \frac{\overline{X} - \mu}{S/\sqrt{n}} $$ has a Student's t-distribution with $n-1$ degrees of freedom, so $$ \overline{X} \pm B_n \frac{S}{\sqrt{n}} $$ where $B_n$ is the corresponding percentage point of Student's distribution, are the endpoints of a corresponding confidence interval if (more realistically) $\sigma$ is unknown.

There is more uncertainty if $\sigma$ is unknown than if $\sigma$ is known, THEREFORE $B_n > A$.

If $\sigma$ is unknown, then the amount of uncertainty decreases as the sample size $n$ increases, THEREFORE $B_n$ decreases as $n$ increases. (Of course, the fact that $B_2 > B_3 > B_4 > \cdots > A$ can be proved by standard arguments involving fiddling with integrals.)

The logical leaps corresponding to the THEREFOREs above have a seeming intellectual compellingness about them. It would be extraordinarily paradoxical if the conclusions following them failed to hold despite the reasoning preceding them. But these are not logically rigorous arguments.

So my question is whether there could be some theorem of logic that says things like those following the THEREFOREs are true when things like those preceding them are true.

  • 1
    $\begingroup$ I don't see where the whole discussion of marbles sampled with or without replacement fit with the rest of the question. $\endgroup$ – Thierry Zell Dec 15 '10 at 21:59
  • $\begingroup$ Do you have a mathematical meaning of "uncertainty" in mind? $\endgroup$ – Daniel Litt Dec 16 '10 at 2:11
  • 1
    $\begingroup$ The inference from the things stated before "THEREFORE" in the situation with the marbles to the conclusion stated after it, is the same sort of thing as the inference corresponding to "THEREFORE" involving the normal distribution. $\endgroup$ – Michael Hardy Dec 16 '10 at 4:32
  • 1
    $\begingroup$ What does this have to do with logic? $\endgroup$ – Felipe Voloch Dec 16 '10 at 16:05
  • 2
    $\begingroup$ @Michael:In a site such as this, I would interpret the logic tag to mean that the question had something to do with Mathematical Logic (the subdiscipline) not that some statement was a logical consequence of another. $\endgroup$ – Felipe Voloch Dec 16 '10 at 19:45

Let's recall how all these confidence intervals work in general. You have some real parameters $t,s$ that define the distribution. You have two unbiased estimators $T$ and $S$ for these parameters that depend on the (vector) observation $X$. You want to create the confidence interval for $t$ from $T$ and known $s$ in one case and from $T$ and $S$ in the other case. We will also assume that the distribution of $T$ is symmetric around $t$ for all $s$ with the density decreasing in both directions, and the distribution of $T-t$ depends on $s$ but not on $t$. The first case is then simple: we take $T$ and find the least $a=a(s)$ such that $P(|T-t|>a(s))<\delta$ and say that $[T-a(s),T+a(s)]$ is the $\delta$-confidence interval. What we mean by that is that regardless of what the actual value of $t$ is, the probability that this interval won't cover it is less than $\delta$. Let's also assume that $s>0$ and $a(s)$ is increasing in $s$ Now, let us ask whether it is possible that $[T-a(S),T+a(S)]$ is a $\delta'$-confidence interval with $\delta'<\delta$ (which, in your terminology, would mean that determining $s$ from observation beats knowing it)?

The answer is, of course "Yes": despite $S$ is an unbiased estimator, it is perfectly possible that $S$ overestimates more often than it underestimates and $a$ is more sensitive to overestimates than to underestimates. To create a particular example, take the uniform distribution on the interval $[t-s^2,t+s^2]$ and the observation $X_1,X_2,X_3$. Take $T=X_1$ (I know, this is a bit stupid, but how do you know whether your estimators are not equally stupid in general?). Then $a(s)=(1-\delta)s^2$. Now, take $S=c\sqrt{|X_2-X_3|}$ where $c$ is chosen so that $S$ is unbiased. It remains to compute the probability that the true value is in the interval. $S$ is independent of $T$ and is $ys$ with probability $p(y)dy$ with some positive continuous $p$ on $[0,c\sqrt{2})$. So, we get $\int p(y)\min(1,(1-\delta)y^2)dy=(1-\delta) \int p(y)y^2dy>(1-\delta)$ because $\int p(y)ydy=1$ already ($S$ is unbiased), provided that $1-\delta$ is small enough (which is ridiculous too, but how do you tell the difference?). So, your $B_n$ may be larger than $A$ not for some deep probabilistic reason, but for the mundane reason that $\sqrt s$ is concave, not convex like $s^2$ in my example.

Now, this doesn't show that we can do better with less information. It merely shows that the proposed way to compare effectiveness is fundamentally flawed. But how else can you compare the confidence intervals?

  • $\begingroup$ For $S$ to be unbiased, $c$ must depend on $s$ (in fact, $c=sc_0$ for an absolute constant $c_0$). Hence $S$ is not an estimator of $s$ based on the sample $X$. But maybe I am missing something? $\endgroup$ – Did Dec 16 '10 at 6:50
  • $\begingroup$ Check again: $X_2-X_3$ is proportional to $s^2$ (stretch the interval, and the difference will stretch the same number of times), so the square root is proportional to $s$, meaning that if you divide it by $s$, the distribution won't depend on $s$. Am I missing something? Note that $S$ is an estimator for the parameter $s$, not for the length $s^2$! $\endgroup$ – fedja Dec 16 '10 at 7:28
  • $\begingroup$ fedja: you are right. Sorry about the noise. $\endgroup$ – Did Dec 16 '10 at 12:24
  • 1
    $\begingroup$ Didier: No need to apologize. It is always pleasant when somebody reads your posts or papers attentively enough to start noticing real or imaginary errors (an imaginary error a reader notices is just a signal that something may not be written clear enough or emphasized, and once a signal carries a message, it cannot be noise) :). $\endgroup$ – fedja Dec 16 '10 at 13:24
  • $\begingroup$ @fedja: I'm slightly rushed today, but I think within the next couple of hours I'll have digested your answer somewhat............ $\endgroup$ – Michael Hardy Dec 16 '10 at 15:32

$\newcommand\si{\sigma}$There is no THEREFORE in general. Indeed, suppose we observe the values of the pair $(X,S)$ of independent random variables, where $P(X=\mu+\si)=P(X=\mu-\si)=1/4$, $P(X=\mu)=1/2$, $S:=\sqrt{S^2}$, $P(S^2=\si^2/2)=P(S^2=3\si^2/2)=1/2$, $\mu$ is an unknown real parameter, and $\si$ is a possibly unknown positive real parameter. If $\si$ is known, our confidence interval (CI) (for $\mu$) is symmetric about $X$ and based on the pivot $\frac{X-\mu}\si$ (by definition, a pivot is any random variable whose distribution does not depend on the parameters). If $\si$ is unknown, our CI is again symmetric about $X$ but now based on the pivot $\frac{X-\mu}S$. This setting is quite similar to your normal-sampling setting, with my pair $(X,S)$ in place of your pair $(\bar X,S)$. In particular, I also have $ES^2=\si^2$. (If desired, one may arbitrarily closely approximate the discrete distributions of $X$ and $S$ by continuous ones.)

Now, the CI with $\si$ known will be always better than the CI with $\si$ unknown if and only if $U:=\big(\frac{X-\mu}\si\big)^2$ is stochastically less than $V:=\big(\frac{X-\mu}S\big)^2$, that is, if $$P(U\ge u)\overset{\text{(?)}}\le P(V\ge u) \tag{1}$$ for all real $u$. However, $$P(U\ge1)=\tfrac12>\tfrac12\times\tfrac12 \\ =P\big((X-\mu)^2=\si^2\big)\,P(S^2=\si^2/2) \\ =P\big((X-\mu)^2=\si^2,S^2=\si^2/2\big) \\ =P(V\ge1).$$ So, (1) fails to hold for $u=1$.

However, $U$ is better than $V$ on an average: $$EV=E\Big(U\frac{\si^2}{S^2}\Big)=EU\,E\Big(\frac{\si^2}{S^2}\Big) >EU\,\frac{\si^2}{ES^2}=EU, $$ where the inequality is an instance of Jensen's inequality.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.