Take the 2-minute tour ×
MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

I have two variables, $x$ and $y$, and I am using linear regression to predict $y$ from $x$ over a large set of subjects. There are multiple observations per subject.

I have tried several things:

  1. pool all observations together and fit one model to the entire dataset;
  2. fit separate models (same specification as (1)) for each subject.

Now, (1) works reasonably well but obviously takes no account of subject-specific effects. On the other hand (2) is prone to overfitting, which manifests itself in poor out-of-sample performance.

If I average the two predictions, the result performs better than both (1) and (2) out of sample. However, this is clearly somewhat ad hoc.

My question is: what might be a better way to combine (1) and (2) into a single predictor?

Also, I have reasons to think that "similar" subjects should have similar regression coefficients. Is there any way to make use of this?

[Edit] I should have mentioned that there is no natural hierarchy to the subjects. However, I think I can come up with a reasonable similarity metric.

share|improve this question
    
You really do not need any hierarchy to the subjects to do hierarchical bayesian estimation. The word 'hierarchical' comes from assuming that the parameters of interest (the \beta_i vectors in your model) come from a normal distribution. –  Anon Jun 2 '10 at 9:57

3 Answers 3

up vote 0 down vote accepted

You should use hierarchical bayesian regression. Google search will provide lots of pointers.

share|improve this answer

As a graduate student I did lots of exercises that were of a kind that this fits into neatly, in which one was to derive the likelihood-ratio test of any of various null hypotheses. Let's see: $$ Y = X\alpha + M\varepsilon + \eta $$ where $$ X = \begin{pmatrix} 1 & x_1 \\ \vdots & \vdots \\ 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \\ \vdots & \vdots \\ 1 & x_n \end{pmatrix}, $$ $$\alpha = \begin{pmatrix} \alpha_0 \\ \alpha_1 \end{pmatrix}$$, $$ M = \begin{pmatrix} 1 \\ \vdots \\ 1 \\ & 1 \\ & \vdots \\ & 1 \\ & & \ddots \\ & & & 1 \\ & & & \vdots \\ & & & 1 \end{pmatrix}, $$ $\varepsilon$ is normally distributed with expected value a column of $n$ zeros and variance $\sigma^2$ times the $n\times n$ identity matrix, and $\eta$ is normally distributed with expected value a column of as many zeros as you have observations and variance $\tau^2$ times a larger identity matrix.

With all that, one can derive maximum likelihood estimates (MLEs) of $\alpha$, $\sigma^2$, and $\tau^2$. And then one finds the probability distributions of the MLEs. Much can be said about the specifics.

Such were the exercises I did, but that's far from the end of the story.......

share|improve this answer

You could try a mixed-effects model, which is a simpler version of the hierarchical Bayes approach suggested above. These models can be fitted very easily in statistics packages. If you don't want to spend much money, and don't mind a learning curve, then R would be a natural place to start.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.