Regression with correlation structure

Question

I have a theoretical question about regression models. Let's say I measured multiple responses from $n$ subjects and these responses are correlated with each other. For example, let's say I measured heart rate and body temperature from $n$ individuals with the following categorical factors: sex (male/female) and age (young/adult).

It's quite possible that heart rate and body temperature are correlated to some extent in any individual.

Ignoring the correlation between heart rate and body temperature I'll model each response independently as:

$y_{heart\_rate} = X\beta_{heart\_rate} + \epsilon_{heart\_rate}$

$y_{body\_temperature} = X\beta_{body\_temperature} + \epsilon_{body\_temperature}$

where $X$ is identical for the two models and is of dimensions $n \times 2$, and:

$\epsilon_{heart\_rate} \sim N(0,\sigma^2_{heart\_rate})$

$\epsilon_{body\_temperature} \sim N(0,\sigma^2_{body\_temperature})$

If I do want to take the correlation into account I would define $X^*$ as a block diagonal matrix of $X$'s. I.e., $X^* = \left( \begin{array}{cc} X & 0 \\ 0 & X \\ \end{array} \right)$

and $\epsilon^* \sim N_2(0,\left( \begin{array}{cc} \sigma^2_{heart\_rate} & \rho \\ \rho & \sigma^2_{body\_temperature} \\ \end{array} \right))$

For simplicity let's assume that $\rho$ (the correlation between heart rate and body temperature) is given.

And my model will be:

$ \left( \begin{array}{c} y_{heart\_rate}\\ y_{body\_temperature}\\ \end{array} \right) = X^* \left( \begin{array}{c} \beta_{heart\_rate}\\ \beta_{body\_temperature}\\ \end{array} \right) + \epsilon^*$

My question is whether $\hat{\beta}_{heart\_rate}$ and $\hat{\beta}_{body\_temperature}$ and their standard errors will be different under the two different models.

Answer 1

Generalised least squares (GLS) is the best linear unbiased estimator (BLUE) for the case with correlations. For an arbitrary weight matrix $W$, the least squares estimate can be written as $$\hat{\beta} = (X^T W X)^{-1} X^T W y ,$$ where $X$ is the full matrix of the linear model $y = X \beta + \varepsilon$. The covariance of the estimator is $$\Sigma_\hat{\beta} = (X^T W X)^{-1} X^T W \Sigma_\varepsilon W^T X (X^T W X)^{-1},$$ where $\Sigma_\varepsilon$ is the true covariance matrix of $\varepsilon$. Now, if you are doing GLS, i.e. $W = \Sigma_\varepsilon^{-1}$, you get the BLUE $\hat{\beta}$ and you can simplify the variance to $$\Sigma_\beta = (X^T \Sigma_\varepsilon^{-1} X)^{-1}.$$ On the other hand, if you ignore the correlations in $\epsilon$, you are using a different (diagonal) $W$ so that you get in general (i) a different $\hat{\beta}$, and (ii) a larger $\Sigma_\hat{\beta}$.