# Inequality-constrained linear-regression, what is the covariance of the estimator?

If you do a linear regression: $||Ax - e ||^2$, where e is iid Gaussian, mean 0 and variance 1, then your answer is $x_{hat} = (A' A)^{-1} (A' * e)$ and the covariance of $x_{hat}$ is $(A' A)^{-1}$

Now, what if I add the linear inequality constraints $Bx > c$? There are algorithms that find the answer for a given $e$, but what is the covariance matrix?

It seems like a non-trivial problem: http://www.gurulib.com/_user_manual_file/pic_1247578519497.pdf

However, the author seems to give up: "A much more interesting problem is to analyze a properly truncated variance-covariance matrix of $b*$. However, it is beyond the scope of this paper."

Of course, I can do a Monte-Carlo simulation, but a closed-form solution would be better. Any hint or reference?

-
Well, the vector $x$ is random right? I agree it's not normal, but there's still a covariance matrix attached to it. –  Tony Bruguier Oct 14 '10 at 4:46

Well, the vector $x$ is random right?

It's a parameter, so therefore fixed (yet unknown): the estimator $\hat x$ is a random variable. I would agree with Brian that a covariance matrix will not be all that useful the constraints will mean that the estimator will tend to concentrate around the edges, where a lot of the asymptotic machinery breaks down.

Personally, I reckon a Bayesian approach would be better, as the inequality constraints can be easily built into the prior.

-
Brian and Simon, Thanks for your answers. If the constraints are very narrow, then the vector $\hat{x}$ will most likely behave like a binary variable. If the constraints are very loose, then it $\hat{x}$ will behave like a normal variable. Sorry, I don't know how you would use Bayesian approach to solve this problem. Could you enlighten me? Also, please note that I am not interested in the pdf of $\hat{x}$, just its covariance matrix. I understand that a canned answer may or may not exist. The paper I reference seems to indicate that in 1976, there was no such answer. –  Tony Bruguier Oct 14 '10 at 14:27
I haven't read it, but this paper gives a method of computation for a Bayesian approach: stat.duke.edu/~scs/Courses/Stat376/Papers/Constraints/… –  simon Oct 14 '10 at 15:13
Thanks for the reference. I have been reading it, but I am not done yet. Just so that I make sure I understand correctly, the approach still uses a random number generator, right? It's just a smart way to go at it. –  Tony Bruguier Oct 19 '10 at 19:01
Yes: the posterior distribution is intractable (that is, it doesn't have a nice analytic density function), so a random number generator (in this case, a Gibbs sampler) is used to obtain a sample, from which we can compute the quantities of interest (mean, credible intervals, etc.) –  simon Oct 21 '10 at 10:19

"Covariance Matrix" doesn't make a lot of sense in this situation, since there's no multivariate normal distribution around the fitted parameters. You could if you wanted look at a non-ellipsoidal confidence region constructed by taking the intersection of the set of feasible parameters and some contour of the likelihood (or Chi^2.)

-