# Combining regressions [closed]

I have two variables, $x$ and $y$, and I am using linear regression to predict $y$ from $x$ over a large set of subjects. There are multiple observations per subject.

I have tried several things:

1. pool all observations together and fit one model to the entire dataset;
2. fit separate models (same specification as (1)) for each subject.

Now, (1) works reasonably well but obviously takes no account of subject-specific effects. On the other hand (2) is prone to overfitting, which manifests itself in poor out-of-sample performance.

If I average the two predictions, the result performs better than both (1) and (2) out of sample. However, this is clearly somewhat ad hoc.

My question is: what might be a better way to combine (1) and (2) into a single predictor?

Also, I have reasons to think that "similar" subjects should have similar regression coefficients. Is there any way to make use of this?

 I should have mentioned that there is no natural hierarchy to the subjects. However, I think I can come up with a reasonable similarity metric.

-

## closed as off-topic by Ricardo Andrade, Yemon Choi, S. Carnahan♦Nov 10 '14 at 0:37

This question appears to be off-topic. The users who voted to close gave this specific reason:

• "This question does not appear to be about research level mathematics within the scope defined in the help center." – Ricardo Andrade, S. Carnahan
If this question can be reworded to fit the rules in the help center, please edit the question.

You really do not need any hierarchy to the subjects to do hierarchical bayesian estimation. The word 'hierarchical' comes from assuming that the parameters of interest (the \beta_i vectors in your model) come from a normal distribution. – vad Jun 2 '10 at 9:57
This question appears to be off-topic because it would now be better placed on stats.SE – Yemon Choi Nov 10 '14 at 0:12

You should use hierarchical bayesian regression. Google search will provide lots of pointers.

-

As a graduate student I did lots of exercises that were of a kind that this fits into neatly, in which one was to derive the likelihood-ratio test of any of various null hypotheses. Let's see: $$Y = X\alpha + M\varepsilon + \eta$$ where $$X = \begin{pmatrix} 1 & x_1 \\ \vdots & \vdots \\ 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \\ \vdots & \vdots \\ 1 & x_n \end{pmatrix},$$ $$\alpha = \begin{pmatrix} \alpha_0 \\ \alpha_1 \end{pmatrix}$$, $$M = \begin{pmatrix} 1 \\ \vdots \\ 1 \\ & 1 \\ & \vdots \\ & 1 \\ & & \ddots \\ & & & 1 \\ & & & \vdots \\ & & & 1 \end{pmatrix},$$ $\varepsilon$ is normally distributed with expected value a column of $n$ zeros and variance $\sigma^2$ times the $n\times n$ identity matrix, and $\eta$ is normally distributed with expected value a column of as many zeros as you have observations and variance $\tau^2$ times a larger identity matrix.

With all that, one can derive maximum likelihood estimates (MLEs) of $\alpha$, $\sigma^2$, and $\tau^2$. And then one finds the probability distributions of the MLEs. Much can be said about the specifics.

Such were the exercises I did, but that's far from the end of the story.......

-

You could try a mixed-effects model, which is a simpler version of the hierarchical Bayes approach suggested above. These models can be fitted very easily in statistics packages. If you don't want to spend much money, and don't mind a learning curve, then R would be a natural place to start.

-