Take the 2-minute tour ×
MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

Hi -- is there any analogue or adjustment of, say, Schwartz Bayesian (or other) information criterion that would be applicable to model selection in ridge regression with a given ridge parameter $\eta$, i.e. $\hat{\beta}_\eta = (X'X+\eta I)^{-1}X'Y$?

Thanks!!

share|improve this question
add comment

1 Answer

up vote 1 down vote accepted

The ridge estimator corresponds to the posterior mean under a Normal linear regression model with a conjugate Normal-inverse-gamma prior on the regression coefficients: $\beta \mid \sigma^2, \lambda \sim \mbox{N}(0, \lambda^{-1}\sigma^2 \mbox{I})$ and $\sigma^2 \sim \mbox{IG}(a,b)$ for known hyperparameters $a$ and $b$. One may additionally put a prior distribution over $\lambda$. If you consider a discrete number of possible values for $\lambda$ then one may compute posterior probabilities for each of these values or compute Bayes factors to compare different values.

As BIC and AIC and other such "information criterions" can be viewed as approximations to Bayes factors, this may answer your question. Usually, as you probably know, one simply checks prediction error for the different values via cross-validation (at least in prediction contexts) and selects lambda that way.

share|improve this answer
    
Thanks! Do I understand it right that in your notation $\lambda$ is the ridge parameter ($\eta$ in the question)? I am a bit confused as to how to use these priors to construct some analogue of AIC or BIC, I am actually concerned with selecting a subset of $X$'s to use. It does look like cross-validation is the way to go, as you suggest. –  laxxy Nov 15 '10 at 2:25
    
OK, I misread the question -- I thought you were asking about using BIC or AIC to select the value of the ridge parameter. For doing subset selection of the predictors I'd look into the lasso estimator rather than ridge regression. There is a whole ton of literature on penalized regression, of which ridge is one flavor and lasso is another. The Elements of Statistical Learning is probably the classical text for these things. –  R Hahn Nov 15 '10 at 3:01
    
Thanks for the references! The situation is this: there is a model that generates data (Y and X). Under some parameter values, some X's may happen to be very close, plus some small noise, likely due to simulation error. Full-sample OLS tries to use this noise, and produces crazy estimates. However, it is almost always the best model in cross-validation. Setting the ridge parameter first and then doing cross-val. to pick components of X seems to work better, but I am not sure if there is a good way to do that. Have to look closer at lasso, first impression so far is that it overfits somewhat. –  laxxy Nov 16 '10 at 14:20
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.