Take the 2-minute tour ×
MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

Is there a univariate probability distribution $p_{\lambda,\alpha}(\beta)$ over the reals, parameterized by $\lambda > 0$ and $1 >= \alpha >= 0$, such that $p_{\lambda,\alpha} \propto \exp(-\lambda(\alpha \beta^{2} + (1 - \alpha)|\beta|)$? If so, is it a proper distribution (integrates to 1 over the real line)? Does the density function have a closed form expression? If not, does the density function have some other nice representation?

Background: One approach to avoiding overfitting in regression modeling with many predictors is to minimize the sum of prediction error plus a penalty based on the size of the coefficients. This is particularly used in machine learning problems with large numbers of features, as in computational linguistics and information retrieval. If an L2 penalty (i.e. a penalty proportional to the square of the coefficient) is used this is called ridge regression. It is equivalent to finding the Bayesian maximum a posteriori (MAP) fit with a prior that is the product of univariate gaussian distributions with mean 0. If an L1 penalty (proportion to absolute value of coefficient) is used this is called lasso regression, and is equivalent to finding the Bayesian MAP fit under a prior which is a product of univariate Laplace (double exponential) distributions with mean 0. This paper:

Zou, Hui and Trevor Hastie. 2005. Regularization and Variable Selection via the Elastic Net. Journal of the Royal Statistical Society B. 67(Part 2):301–320.

introduced the "elastic net" which is a weighted combination of L1 and L2 penalties. My question is whether the elastic net corresponds to MAP estimation under a prior which is a product of univariate distributions, and if so what the nature of that distribution is.

share|improve this question
add comment

2 Answers

up vote 0 down vote accepted

So here's my attempt at the algebra to find the normalizing constant:

$Z_{\lambda,\alpha} =\displaystyle\int \exp( -\lambda(\alpha x^2 + (1 - \alpha)|x| ) ) dx$

$ = 2 \displaystyle\int^{0}_{-\infty} \exp( -\lambda(\alpha x^2 + (1 - \alpha)x ) ) dx$

$ = 2 \displaystyle\int^{0}_{-\infty} \exp( -\lambda \alpha x^2 -\lambda (1 - \alpha)x) dx$

$ = 2 \displaystyle\int^{0}_{-\infty} \exp(-ax^2-bx) dx$

where $a = \lambda \alpha$ and $b = \lambda (1 - \alpha)$. From Mathematica I get

$\displaystyle\int \exp(-ax^2-bx) dx = \frac{\sqrt{\pi}}{2 \sqrt{a}} \exp \left( \frac{b^2}{4a} \right) \mbox{erf} \left( \frac{2ax+b}{2 \sqrt{a}} \right)$

So continuing

$Z_{\lambda,\alpha} = \left. 2 \dfrac{\sqrt{\pi}}{2 \sqrt{a}} \exp \left( \dfrac{b^2}{4a} \right) \mbox{erf} \left( \dfrac{2ax+b}{2 \sqrt{a}} \right) \right|^{0}_{-\infty}$

$ = \left. \dfrac{\sqrt{\pi}}{\sqrt{a}} \exp \left( \dfrac{b^2}{4a} \right) \left( 2\Phi \left( \sqrt{2} \dfrac{2ax+b}{2 \sqrt{a}} \right) - 1 \right) \right|^{0}_{-\infty}$

$ = \left. \dfrac{\sqrt{\pi}}{\sqrt{a}} \exp \left( \dfrac{b^2}{4a} \right) \left( 2\Phi \left( \dfrac{2ax+b}{ \sqrt{2a}} \right) - 1 \right) \right|^{0}_{-\infty}$

$ = \left. \dfrac{\sqrt{\pi}}{\sqrt{a}} \exp \left( \dfrac{b^2}{4a} \right) \left( 2\Phi \left( \dfrac{2ax+b}{ \sqrt{2a}} \right) - 1 \right) \right|^{0}_{-\infty}$

$ = 2 \dfrac{\sqrt{\pi}}{\sqrt{a}} \exp \left( \dfrac{b^2}{4a} \right) \Phi \left( \dfrac{b}{ \sqrt{2a}} \right) $

$ = 2 \dfrac{\sqrt{\pi}}{\sqrt{\alpha \lambda}} \exp \left( \dfrac{\lambda^2(1-\alpha)^2}{4\lambda \alpha} \right) \Phi \left( \dfrac{\lambda(1-\alpha)}{ \sqrt{2\lambda \alpha}} \right) $

$ = 2 \dfrac{\sqrt{\pi}}{\sqrt{\alpha \lambda}} \exp \left( \dfrac{\lambda(1-\alpha)^2}{4 \alpha} \right) \Phi \left( \dfrac{\sqrt{\lambda}(1-\alpha)}{ \sqrt{2 \alpha}} \right) $

Am I going wrong somewhere?

share|improve this answer
add comment

I don't know if that particular density has a name, but it is a valid exponential family density. The normalisation constant does not have a closed form expression, but with a bit of work you can write as a function of the cumulative Gaussian density, by noting that, since the density is symmetrical around 0: $\int{\exp(-\lambda(\alpha x^2 + (1-\alpha) |x|))}dx$=$2\int_{-\infty}^0{\exp(Ax^2+Bx)}$

After some algebra you find, for $\alpha > 0$:

$\int{\exp(-\lambda(\alpha x^2 + (1-\alpha) |x|))}dx$=$2\Phi(0;(1-\alpha)/2\alpha,1/(2\alpha))\exp\(\lambda/(4\alpha(1-\alpha)^2)\)\sqrt{\pi/(\alpha\lambda)}$

That's finite for $\alpha,\lambda>0$ (of course for $\alpha=0$ you just recover the Laplace density so it works too). As far as I can tell the elastic net is a valid MAP estimate.

share|improve this answer
    
Thanks! I should have thought about using the symmetry that way. I'm not familiar, however, with the three parameter notation you're using for the cumulative normal distribution function. And when I try to crank through the algebra I also get a different quantity inside exp(). I can't seem to get LaTex to work inside a comment, so I'll show my algebra in an answer - see below. –  daviddlewis Jan 21 '10 at 16:11
    
Sorry, I made a mistake copying down the formula, there's a factor missing in the exponential. The $\Phi(x,\mu,\sigma^2)$ notation means "cumulative gaussian distribution at x, for a Gaussian with mean \mu and variance \sigma^2" –  Simon Barthelmé Jan 22 '10 at 16:17
    
So does the version in my answer look right to you now? –  daviddlewis Feb 18 '10 at 17:05
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.