1
$\begingroup$

If you do a linear regression: $||Ax - e ||^2$, where e is iid Gaussian, mean 0 and variance 1, then your answer is $x_{hat} = (A' A)^{-1} (A' * e)$ and the covariance of $x_{hat}$ is $(A' A)^{-1}$

Now, what if I add the linear inequality constraints $Bx > c$? There are algorithms that find the answer for a given $e$, but what is the covariance matrix?

It seems like a non-trivial problem: http://www.gurulib.com/_user_manual_file/pic_1247578519497.pdf

However, the author seems to give up: "A much more interesting problem is to analyze a properly truncated variance-covariance matrix of $b*$. However, it is beyond the scope of this paper."

Of course, I can do a Monte-Carlo simulation, but a closed-form solution would be better. Any hint or reference?

Thanks in advance, Tony

Improve this question
$\endgroup$
1
  • $\begingroup$ Well, the vector $x$ is random right? I agree it's not normal, but there's still a covariance matrix attached to it. $\endgroup$
    – Tony Bruguier
    Commented Oct 14, 2010 at 4:46

2 Answers 2

Reset to default
1
$\begingroup$

Well, the vector $x$ is random right?

It's a parameter, so therefore fixed (yet unknown): the estimator $\hat x$ is a random variable. I would agree with Brian that a covariance matrix will not be all that useful the constraints will mean that the estimator will tend to concentrate around the edges, where a lot of the asymptotic machinery breaks down.

Personally, I reckon a Bayesian approach would be better, as the inequality constraints can be easily built into the prior.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Brian and Simon, Thanks for your answers. If the constraints are very narrow, then the vector $\hat{x}$ will most likely behave like a binary variable. If the constraints are very loose, then it $\hat{x}$ will behave like a normal variable. Sorry, I don't know how you would use Bayesian approach to solve this problem. Could you enlighten me? Also, please note that I am not interested in the pdf of $\hat{x}$, just its covariance matrix. I understand that a canned answer may or may not exist. The paper I reference seems to indicate that in 1976, there was no such answer. $\endgroup$
    – Tony Bruguier
    Commented Oct 14, 2010 at 14:27
  • $\begingroup$ I haven't read it, but this paper gives a method of computation for a Bayesian approach: stat.duke.edu/~scs/Courses/Stat376/Papers/Constraints/… $\endgroup$
    – simon
    Commented Oct 14, 2010 at 15:13
  • $\begingroup$ Thanks for the reference. I have been reading it, but I am not done yet. Just so that I make sure I understand correctly, the approach still uses a random number generator, right? It's just a smart way to go at it. $\endgroup$
    – Tony Bruguier
    Commented Oct 19, 2010 at 19:01
  • $\begingroup$ Yes: the posterior distribution is intractable (that is, it doesn't have a nice analytic density function), so a random number generator (in this case, a Gibbs sampler) is used to obtain a sample, from which we can compute the quantities of interest (mean, credible intervals, etc.) $\endgroup$
    – simon
    Commented Oct 21, 2010 at 10:19
0
$\begingroup$

"Covariance Matrix" doesn't make a lot of sense in this situation, since there's no multivariate normal distribution around the fitted parameters. You could if you wanted look at a non-ellipsoidal confidence region constructed by taking the intersection of the set of feasible parameters and some contour of the likelihood (or Chi^2.)

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .