Combining regressions [closed]

Question

I have two variables, $x$ and $y$, and I am using linear regression to predict $y$ from $x$ over a large set of subjects. There are multiple observations per subject.

I have tried several things:

1. pool all observations together and fit one model to the entire dataset;
2. fit separate models (same specification as (1)) for each subject.

Now, (1) works reasonably well but obviously takes no account of subject-specific effects. On the other hand (2) is prone to overfitting, which manifests itself in poor out-of-sample performance.

If I average the two predictions, the result performs better than both (1) and (2) out of sample. However, this is clearly somewhat ad hoc.

My question is: what might be a better way to combine (1) and (2) into a single predictor?

Also, I have reasons to think that "similar" subjects should have similar regression coefficients. Is there any way to make use of this?

[Edit] I should have mentioned that there is no natural hierarchy to the subjects. However, I think I can come up with a reasonable similarity metric.

Answer 1

You should use hierarchical bayesian regression. Google search will provide lots of pointers.

Answer 2

As a graduate student I did lots of exercises that were of a kind that this fits into neatly, in which one was to derive the likelihood-ratio test of any of various null hypotheses. Let's see: $$ Y = X\alpha + M\varepsilon + \eta $$ where $$ X = \begin{pmatrix} 1 & x_1 \\ \vdots & \vdots \\ 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \\ \vdots & \vdots \\ 1 & x_n \end{pmatrix}, $$ $$\alpha = \begin{pmatrix} \alpha_0 \\ \alpha_1 \end{pmatrix}$$, $$ M = \begin{pmatrix} 1 \\ \vdots \\ 1 \\ & 1 \\ & \vdots \\ & 1 \\ & & \ddots \\ & & & 1 \\ & & & \vdots \\ & & & 1 \end{pmatrix}, $$ $\varepsilon$ is normally distributed with expected value a column of $n$ zeros and variance $\sigma^2$ times the $n\times n$ identity matrix, and $\eta$ is normally distributed with expected value a column of as many zeros as you have observations and variance $\tau^2$ times a larger identity matrix.

With all that, one can derive maximum likelihood estimates (MLEs) of $\alpha$, $\sigma^2$, and $\tau^2$. And then one finds the probability distributions of the MLEs. Much can be said about the specifics.

Such were the exercises I did, but that's far from the end of the story.......

Answer 3

You could try a mixed-effects model, which is a simpler version of the hierarchical Bayes approach suggested above. These models can be fitted very easily in statistics packages. If you don't want to spend much money, and don't mind a learning curve, then R would be a natural place to start.