MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have two variables, $x$ and $y$, and I am using linear regression to predict $y$ from $x$ over a large set of subjects. There are multiple observations per subject.

I have tried several things:

  1. pool all observations together and fit one model to the entire dataset;
  2. fit separate models (same specification as (1)) for each subject.

Now, (1) works reasonably well but obviously takes no account of subject-specific effects. On the other hand (2) is prone to overfitting, which manifests itself in poor out-of-sample performance.

If I average the two predictions, the result performs better than both (1) and (2) out of sample. However, this is clearly somewhat ad hoc.

My question is: what might be a better way to combine (1) and (2) into a single predictor?

Also, I have reasons to think that "similar" subjects should have similar regression coefficients. Is there any way to make use of this?

[Edit] I should have mentioned that there is no natural hierarchy to the subjects. However, I think I can come up with a reasonable similarity metric.

share|cite|improve this question

closed as off-topic by Ricardo Andrade, Yemon Choi, S. Carnahan Nov 10 '14 at 0:37

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question does not appear to be about research level mathematics within the scope defined in the help center." – Ricardo Andrade, S. Carnahan
If this question can be reworded to fit the rules in the help center, please edit the question.

You really do not need any hierarchy to the subjects to do hierarchical bayesian estimation. The word 'hierarchical' comes from assuming that the parameters of interest (the \beta_i vectors in your model) come from a normal distribution. – vad Jun 2 '10 at 9:57
This question appears to be off-topic because it would now be better placed on stats.SE – Yemon Choi Nov 10 '14 at 0:12
up vote 0 down vote accepted

You should use hierarchical bayesian regression. Google search will provide lots of pointers.

share|cite|improve this answer

As a graduate student I did lots of exercises that were of a kind that this fits into neatly, in which one was to derive the likelihood-ratio test of any of various null hypotheses. Let's see: $$ Y = X\alpha + M\varepsilon + \eta $$ where $$ X = \begin{pmatrix} 1 & x_1 \\ \vdots & \vdots \\ 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \\ \vdots & \vdots \\ 1 & x_n \end{pmatrix}, $$ $$\alpha = \begin{pmatrix} \alpha_0 \\ \alpha_1 \end{pmatrix}$$, $$ M = \begin{pmatrix} 1 \\ \vdots \\ 1 \\ & 1 \\ & \vdots \\ & 1 \\ & & \ddots \\ & & & 1 \\ & & & \vdots \\ & & & 1 \end{pmatrix}, $$ $\varepsilon$ is normally distributed with expected value a column of $n$ zeros and variance $\sigma^2$ times the $n\times n$ identity matrix, and $\eta$ is normally distributed with expected value a column of as many zeros as you have observations and variance $\tau^2$ times a larger identity matrix.

With all that, one can derive maximum likelihood estimates (MLEs) of $\alpha$, $\sigma^2$, and $\tau^2$. And then one finds the probability distributions of the MLEs. Much can be said about the specifics.

Such were the exercises I did, but that's far from the end of the story.......

share|cite|improve this answer

You could try a mixed-effects model, which is a simpler version of the hierarchical Bayes approach suggested above. These models can be fitted very easily in statistics packages. If you don't want to spend much money, and don't mind a learning curve, then R would be a natural place to start.

share|cite|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.