3
$\begingroup$

I have a theoretical question about regression models. Let's say I measured multiple responses from $n$ subjects and these responses are correlated with each other. For example, let's say I measured heart rate and body temperature from $n$ individuals with the following categorical factors: sex (male/female) and age (young/adult).

It's quite possible that heart rate and body temperature are correlated to some extent in any individual.

Ignoring the correlation between heart rate and body temperature I'll model each response independently as:

$y_{heart\_rate} = X\beta_{heart\_rate} + \epsilon_{heart\_rate}$

$y_{body\_temperature} = X\beta_{body\_temperature} + \epsilon_{body\_temperature}$

where $X$ is identical for the two models and is of dimensions $n \times 2$, and:

$\epsilon_{heart\_rate} \sim N(0,\sigma^2_{heart\_rate})$

$\epsilon_{body\_temperature} \sim N(0,\sigma^2_{body\_temperature})$

If I do want to take the correlation into account I would define $X^*$ as a block diagonal matrix of $X$'s. I.e., $X^* = \left( \begin{array}{cc} X & 0 \\ 0 & X \\ \end{array} \right)$

and $\epsilon^* \sim N_2(0,\left( \begin{array}{cc} \sigma^2_{heart\_rate} & \rho \\ \rho & \sigma^2_{body\_temperature} \\ \end{array} \right))$

For simplicity let's assume that $\rho$ (the correlation between heart rate and body temperature) is given.

And my model will be:

$ \left( \begin{array}{c} y_{heart\_rate}\\ y_{body\_temperature}\\ \end{array} \right) = X^* \left( \begin{array}{c} \beta_{heart\_rate}\\ \beta_{body\_temperature}\\ \end{array} \right) + \epsilon^*$

My question is whether $\hat{\beta}_{heart\_rate}$ and $\hat{\beta}_{body\_temperature}$ and their standard errors will be different under the two different models.

$\endgroup$
  • $\begingroup$ stats.stackexchange.com may be more relevant $\endgroup$ – Suvrit Oct 9 '15 at 21:06
  • $\begingroup$ en.wikipedia.org/wiki/Seemingly_unrelated_regressions $\endgroup$ – R Hahn May 6 '19 at 2:18
  • $\begingroup$ I think the model makes no sense. The heart rate you are measuring cannot appear in both the response and the predictor. This does not work well unless you are working with time series data. Also if you know heart rate and body temperature are correlated, it should be rather routine to get a LSE of the correlation coefficient. You can then do regression again based on the new variables if you wish. $\endgroup$ – Bombyx mori May 6 '19 at 2:34
1
$\begingroup$

Generalised least squares (GLS) is the best linear unbiased estimator (BLUE) for the case with correlations. For an arbitrary weight matrix $W$, the least squares estimate can be written as $$\hat{\beta} = (X^T W X)^{-1} X^T W y ,$$ where $X$ is the full matrix of the linear model $y = X \beta + \varepsilon$. The covariance of the estimator is $$\Sigma_\hat{\beta} = (X^T W X)^{-1} X^T W \Sigma_\varepsilon W^T X (X^T W X)^{-1},$$ where $\Sigma_\varepsilon$ is the true covariance matrix of $\varepsilon$. Now, if you are doing GLS, i.e. $W = \Sigma_\varepsilon^{-1}$, you get the BLUE $\hat{\beta}$ and you can simplify the variance to $$\Sigma_\beta = (X^T \Sigma_\varepsilon^{-1} X)^{-1}.$$ On the other hand, if you ignore the correlations in $\epsilon$, you are using a different (diagonal) $W$ so that you get in general (i) a different $\hat{\beta}$, and (ii) a larger $\Sigma_\hat{\beta}$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.