0
$\begingroup$

Hi -- is there any analogue or adjustment of, say, Schwartz Bayesian (or other) information criterion that would be applicable to model selection in ridge regression with a given ridge parameter $\eta$, i.e. $\hat{\beta}_\eta = (X'X+\eta I)^{-1}X'Y$?

Thanks!!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The ridge estimator corresponds to the posterior mean under a Normal linear regression model with a conjugate Normal-inverse-gamma prior on the regression coefficients: $\beta \mid \sigma^2, \lambda \sim \mbox{N}(0, \lambda^{-1}\sigma^2 \mbox{I})$ and $\sigma^2 \sim \mbox{IG}(a,b)$ for known hyperparameters $a$ and $b$. One may additionally put a prior distribution over $\lambda$. If you consider a discrete number of possible values for $\lambda$ then one may compute posterior probabilities for each of these values or compute Bayes factors to compare different values.

As BIC and AIC and other such "information criterions" can be viewed as approximations to Bayes factors, this may answer your question. Usually, as you probably know, one simply checks prediction error for the different values via cross-validation (at least in prediction contexts) and selects lambda that way.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks! Do I understand it right that in your notation $\lambda$ is the ridge parameter ($\eta$ in the question)? I am a bit confused as to how to use these priors to construct some analogue of AIC or BIC, I am actually concerned with selecting a subset of $X$'s to use. It does look like cross-validation is the way to go, as you suggest. $\endgroup$
    – laxxy
    Commented Nov 15, 2010 at 2:25
  • $\begingroup$ OK, I misread the question -- I thought you were asking about using BIC or AIC to select the value of the ridge parameter. For doing subset selection of the predictors I'd look into the lasso estimator rather than ridge regression. There is a whole ton of literature on penalized regression, of which ridge is one flavor and lasso is another. The Elements of Statistical Learning is probably the classical text for these things. $\endgroup$
    – R Hahn
    Commented Nov 15, 2010 at 3:01
  • $\begingroup$ Thanks for the references! The situation is this: there is a model that generates data (Y and X). Under some parameter values, some X's may happen to be very close, plus some small noise, likely due to simulation error. Full-sample OLS tries to use this noise, and produces crazy estimates. However, it is almost always the best model in cross-validation. Setting the ridge parameter first and then doing cross-val. to pick components of X seems to work better, but I am not sure if there is a good way to do that. Have to look closer at lasso, first impression so far is that it overfits somewhat. $\endgroup$
    – laxxy
    Commented Nov 16, 2010 at 14:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .