MathOverflow is a question and answer site for professional mathematicians. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

If you do a linear regression: $||Ax - e ||^2$, where e is iid Gaussian, mean 0 and variance 1, then your answer is $x_{hat} = (A' A)^{-1} (A' * e)$ and the covariance of $x_{hat}$ is $(A' A)^{-1}$

Now, what if I add the linear inequality constraints $Bx > c$? There are algorithms that find the answer for a given $e$, but what is the covariance matrix?

It seems like a non-trivial problem:

However, the author seems to give up: "A much more interesting problem is to analyze a properly truncated variance-covariance matrix of $b*$. However, it is beyond the scope of this paper."

Of course, I can do a Monte-Carlo simulation, but a closed-form solution would be better. Any hint or reference?

Thanks in advance, Tony

share|cite|improve this question
Well, the vector $x$ is random right? I agree it's not normal, but there's still a covariance matrix attached to it. – Tony Bruguier Oct 14 '10 at 4:46
up vote 1 down vote accepted

Well, the vector $x$ is random right?

It's a parameter, so therefore fixed (yet unknown): the estimator $\hat x$ is a random variable. I would agree with Brian that a covariance matrix will not be all that useful the constraints will mean that the estimator will tend to concentrate around the edges, where a lot of the asymptotic machinery breaks down.

Personally, I reckon a Bayesian approach would be better, as the inequality constraints can be easily built into the prior.

share|cite|improve this answer
Brian and Simon, Thanks for your answers. If the constraints are very narrow, then the vector $\hat{x}$ will most likely behave like a binary variable. If the constraints are very loose, then it $\hat{x}$ will behave like a normal variable. Sorry, I don't know how you would use Bayesian approach to solve this problem. Could you enlighten me? Also, please note that I am not interested in the pdf of $\hat{x}$, just its covariance matrix. I understand that a canned answer may or may not exist. The paper I reference seems to indicate that in 1976, there was no such answer. – Tony Bruguier Oct 14 '10 at 14:27
I haven't read it, but this paper gives a method of computation for a Bayesian approach:… – simon Oct 14 '10 at 15:13
Thanks for the reference. I have been reading it, but I am not done yet. Just so that I make sure I understand correctly, the approach still uses a random number generator, right? It's just a smart way to go at it. – Tony Bruguier Oct 19 '10 at 19:01
Yes: the posterior distribution is intractable (that is, it doesn't have a nice analytic density function), so a random number generator (in this case, a Gibbs sampler) is used to obtain a sample, from which we can compute the quantities of interest (mean, credible intervals, etc.) – simon Oct 21 '10 at 10:19

"Covariance Matrix" doesn't make a lot of sense in this situation, since there's no multivariate normal distribution around the fitted parameters. You could if you wanted look at a non-ellipsoidal confidence region constructed by taking the intersection of the set of feasible parameters and some contour of the likelihood (or Chi^2.)

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.