User madhadron - MathOverflow

Answer by madhadron for Interpretations among t-test, ANOVA, Tukey HSD results?

2010-07-13T14:21:19Z

It is very often the case that some subset of your data will come out to be statistically significant by random chance. If you are running t-tests among the levels given the level of each other factor, that's four tests. Your chance of one of those four comparisons being significant at $\alpha=0.05$ is $1 - (0.95)^4$, or about 20%.

The ANOVA, F-test, and HSD all account for this kind of multiple comparison. It's what they were designed to handle.

It's actually quite a nice example of why multiple comparisons are important to take into account.

Answer by madhadron for Statistics for mathematicians

2010-07-13T14:16:01Z

Kiefer's "Introduction to Statistical Inference" is a particularly nice book. It crams an incredibly amount of perspective into a very easy to read package. You'll want to supplement it with volumes like Lehmann's "Point Estimation" and "Testing Statistical Hypotheses."

One thing I particularly like is that he starts at the bottom with the decision theory structure, and imposing other criteria to be able to choose from the plethora of admissible procedures before diving into things like linear estimators and hypothesis tests.

Answer by madhadron for statistical approach to multinomial distribution

2010-07-13T14:09:54Z

The classical approach is to build a Neyman-Pearson style hypothesis test (warning: incredibly ugly mathematics, in desperate need of replacement, but ubiquitous).

Say you rolled your die $N$ times to produce $X$. Let the multinomial distribution have parameters $(p_1, p_2, ..., p_6)$, where $\sum_i p_i = 1$. Then construct a one dimensional measure such as $Q = \| X/N - p \|$, using your favorite $p$-norm. Calculate the probability distribution of $Q$.

Your null hypothesis in this case is $p_i = \frac{1}{6}$ for all $i$. For a test of level of significance $\alpha$ (conventionally 0.05 or 0.01), there is a region $[a,b]$ such that $\int_a^b p(Q = x) dx = 1 - \alpha$. Actually, there are many such, and there are other criteria to choose among them. In your case, invariance might be a good one: you expect the whole problem to be symmetric if you let $Q$ go to $-Q$, in which case the interval should be symmetric about 0, i.e., $[-a,a]$.

For a given value of $Q$ from your data, you do the integral over $[-Q,Q]$ and get $1 - \alpha$. That $\alpha$ is the lowest level of significance at which the observed data will be significant.

As I said, classical hypothesis testing is a very ugly theory. There are other approaches, such as minimax tests which you can construct via Bayes priors, since the set of all Bayes priors contains but is usually not much larger than the set of all admissible statistical procedures.

Answer by madhadron for Medium-Sized Calculations and Organization

2010-06-24T13:30:48Z

I agree with the others that a CAS is probably your best bet, but failing that, the folks who do really sprawling but straightforwards calculations are the theoretical computer scientists. Most of them have adopted a proof notation due to Dijkstra of the form

expression1

= { comment about why equality holds }

expression2

= { comment... }

...

The classic book on this is Dijkstra's "Predicate Calculus and Program Semantics," which is lovely but tends to drive a lot of logicians batty. You can also find nice examples of its use in the small scale in papers like A tutorial on the universality and expressiveness of fold

Comment by madhadron

2010-08-11T14:51:57Z

It is the chance to be declared significant falsely, otherwise we would be worried about $\beta$, not $\alpha$. Neyman-Pearson hypothesis testing is based on fixing $\alpha$ and letting everything else fall where it will. From a decision theory point of view, it is absolutely correct to say "statistically significant by random chance." For decisions D0 and D1 corresponding to hypotheses H0 and H1, we acquire some data, and apply some procedure to get D0 or D1. There is a probability of getting D1 when H0 is true.