0
$\begingroup$

Is there a univariate probability distribution $p_{\lambda,\alpha}(\beta)$ over the reals, parameterized by $\lambda > 0$ and $1 >= \alpha >= 0$, such that $p_{\lambda,\alpha} \propto \exp(-\lambda(\alpha \beta^{2} + (1 - \alpha)|\beta|)$? If so, is it a proper distribution (integrates to 1 over the real line)? Does the density function have a closed form expression? If not, does the density function have some other nice representation?

Background: One approach to avoiding overfitting in regression modeling with many predictors is to minimize the sum of prediction error plus a penalty based on the size of the coefficients. This is particularly used in machine learning problems with large numbers of features, as in computational linguistics and information retrieval. If an L2 penalty (i.e. a penalty proportional to the square of the coefficient) is used this is called ridge regression. It is equivalent to finding the Bayesian maximum a posteriori (MAP) fit with a prior that is the product of univariate gaussian distributions with mean 0. If an L1 penalty (proportion to absolute value of coefficient) is used this is called lasso regression, and is equivalent to finding the Bayesian MAP fit under a prior which is a product of univariate Laplace (double exponential) distributions with mean 0. This paper:

Zou, Hui and Trevor Hastie. 2005. Regularization and Variable Selection via the Elastic Net. Journal of the Royal Statistical Society B. 67(Part 2):301–320.

introduced the "elastic net" which is a weighted combination of L1 and L2 penalties. My question is whether the elastic net corresponds to MAP estimation under a prior which is a product of univariate distributions, and if so what the nature of that distribution is.

$\endgroup$

2 Answers 2

1
$\begingroup$

I don't know if that particular density has a name, but it is a valid exponential family density. The normalisation constant does not have a closed form expression, but with a bit of work you can write as a function of the cumulative Gaussian density, by noting that, since the density is symmetrical around 0: $\int{\exp(-\lambda(\alpha x^2 + (1-\alpha) |x|))}dx$=$2\int_{-\infty}^0{\exp(Ax^2+Bx)}$

After some algebra you find, for $\alpha > 0$:

$\int{\exp(-\lambda(\alpha x^2 + (1-\alpha) |x|))}dx$=$2\Phi(0;(1-\alpha)/2\alpha,1/(2\alpha))\exp\(\lambda/(4\alpha(1-\alpha)^2)\)\sqrt{\pi/(\alpha\lambda)}$

That's finite for $\alpha,\lambda>0$ (of course for $\alpha=0$ you just recover the Laplace density so it works too). As far as I can tell the elastic net is a valid MAP estimate.

$\endgroup$
3
  • $\begingroup$ Thanks! I should have thought about using the symmetry that way. I'm not familiar, however, with the three parameter notation you're using for the cumulative normal distribution function. And when I try to crank through the algebra I also get a different quantity inside exp(). I can't seem to get LaTex to work inside a comment, so I'll show my algebra in an answer - see below. $\endgroup$ Jan 21, 2010 at 16:11
  • $\begingroup$ Sorry, I made a mistake copying down the formula, there's a factor missing in the exponential. The $\Phi(x,\mu,\sigma^2)$ notation means "cumulative gaussian distribution at x, for a Gaussian with mean \mu and variance \sigma^2" $\endgroup$ Jan 22, 2010 at 16:17
  • $\begingroup$ So does the version in my answer look right to you now? $\endgroup$ Feb 18, 2010 at 17:05
0
$\begingroup$

So here's my attempt at the algebra to find the normalizing constant:

$Z_{\lambda,\alpha} =\displaystyle\int \exp( -\lambda(\alpha x^2 + (1 - \alpha)|x| ) ) dx$

$ = 2 \displaystyle\int^{0}_{-\infty} \exp( -\lambda(\alpha x^2 + (1 - \alpha)x ) ) dx$

$ = 2 \displaystyle\int^{0}_{-\infty} \exp( -\lambda \alpha x^2 -\lambda (1 - \alpha)x) dx$

$ = 2 \displaystyle\int^{0}_{-\infty} \exp(-ax^2-bx) dx$

where $a = \lambda \alpha$ and $b = \lambda (1 - \alpha)$. From Mathematica I get

$\displaystyle\int \exp(-ax^2-bx) dx = \frac{\sqrt{\pi}}{2 \sqrt{a}} \exp \left( \frac{b^2}{4a} \right) \mbox{erf} \left( \frac{2ax+b}{2 \sqrt{a}} \right)$

So continuing

$Z_{\lambda,\alpha} = \left. 2 \dfrac{\sqrt{\pi}}{2 \sqrt{a}} \exp \left( \dfrac{b^2}{4a} \right) \mbox{erf} \left( \dfrac{2ax+b}{2 \sqrt{a}} \right) \right|^{0}_{-\infty}$

$ = \left. \dfrac{\sqrt{\pi}}{\sqrt{a}} \exp \left( \dfrac{b^2}{4a} \right) \left( 2\Phi \left( \sqrt{2} \dfrac{2ax+b}{2 \sqrt{a}} \right) - 1 \right) \right|^{0}_{-\infty}$

$ = \left. \dfrac{\sqrt{\pi}}{\sqrt{a}} \exp \left( \dfrac{b^2}{4a} \right) \left( 2\Phi \left( \dfrac{2ax+b}{ \sqrt{2a}} \right) - 1 \right) \right|^{0}_{-\infty}$

$ = \left. \dfrac{\sqrt{\pi}}{\sqrt{a}} \exp \left( \dfrac{b^2}{4a} \right) \left( 2\Phi \left( \dfrac{2ax+b}{ \sqrt{2a}} \right) - 1 \right) \right|^{0}_{-\infty}$

$ = 2 \dfrac{\sqrt{\pi}}{\sqrt{a}} \exp \left( \dfrac{b^2}{4a} \right) \Phi \left( \dfrac{b}{ \sqrt{2a}} \right) $

$ = 2 \dfrac{\sqrt{\pi}}{\sqrt{\alpha \lambda}} \exp \left( \dfrac{\lambda^2(1-\alpha)^2}{4\lambda \alpha} \right) \Phi \left( \dfrac{\lambda(1-\alpha)}{ \sqrt{2\lambda \alpha}} \right) $

$ = 2 \dfrac{\sqrt{\pi}}{\sqrt{\alpha \lambda}} \exp \left( \dfrac{\lambda(1-\alpha)^2}{4 \alpha} \right) \Phi \left( \dfrac{\sqrt{\lambda}(1-\alpha)}{ \sqrt{2 \alpha}} \right) $

Am I going wrong somewhere?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.