univariate prior corresponding to weighted sum of L1 and L2 penalties?

Question

Is there a univariate probability distribution $p_{\lambda,\alpha}(\beta)$ over the reals, parameterized by $\lambda > 0$ and $1 >= \alpha >= 0$, such that $p_{\lambda,\alpha} \propto \exp(-\lambda(\alpha \beta^{2} + (1 - \alpha)|\beta|)$? If so, is it a proper distribution (integrates to 1 over the real line)? Does the density function have a closed form expression? If not, does the density function have some other nice representation?

Background: One approach to avoiding overfitting in regression modeling with many predictors is to minimize the sum of prediction error plus a penalty based on the size of the coefficients. This is particularly used in machine learning problems with large numbers of features, as in computational linguistics and information retrieval. If an L2 penalty (i.e. a penalty proportional to the square of the coefficient) is used this is called ridge regression. It is equivalent to finding the Bayesian maximum a posteriori (MAP) fit with a prior that is the product of univariate gaussian distributions with mean 0. If an L1 penalty (proportion to absolute value of coefficient) is used this is called lasso regression, and is equivalent to finding the Bayesian MAP fit under a prior which is a product of univariate Laplace (double exponential) distributions with mean 0. This paper:

Zou, Hui and Trevor Hastie. 2005. Regularization and Variable Selection via the Elastic Net. Journal of the Royal Statistical Society B. 67(Part 2):301–320.

introduced the "elastic net" which is a weighted combination of L1 and L2 penalties. My question is whether the elastic net corresponds to MAP estimation under a prior which is a product of univariate distributions, and if so what the nature of that distribution is.

Answer 1

I don't know if that particular density has a name, but it is a valid exponential family density. The normalisation constant does not have a closed form expression, but with a bit of work you can write as a function of the cumulative Gaussian density, by noting that, since the density is symmetrical around 0: $\int{\exp(-\lambda(\alpha x^2 + (1-\alpha) |x|))}dx$=$2\int_{-\infty}^0{\exp(Ax^2+Bx)}$

After some algebra you find, for $\alpha > 0$:

$\int{\exp(-\lambda(\alpha x^2 + (1-\alpha) |x|))}dx$=$2\Phi(0;(1-\alpha)/2\alpha,1/(2\alpha))\exp\(\lambda/(4\alpha(1-\alpha)^2)\)\sqrt{\pi/(\alpha\lambda)}$

That's finite for $\alpha,\lambda>0$ (of course for $\alpha=0$ you just recover the Laplace density so it works too). As far as I can tell the elastic net is a valid MAP estimate.

Answer 2

So here's my attempt at the algebra to find the normalizing constant:

$Z_{\lambda,\alpha} =\displaystyle\int \exp( -\lambda(\alpha x^2 + (1 - \alpha)|x| ) ) dx$

$ = 2 \displaystyle\int^{0}_{-\infty} \exp( -\lambda(\alpha x^2 + (1 - \alpha)x ) ) dx$

$ = 2 \displaystyle\int^{0}_{-\infty} \exp( -\lambda \alpha x^2 -\lambda (1 - \alpha)x) dx$

$ = 2 \displaystyle\int^{0}_{-\infty} \exp(-ax^2-bx) dx$

where $a = \lambda \alpha$ and $b = \lambda (1 - \alpha)$. From Mathematica I get

$\displaystyle\int \exp(-ax^2-bx) dx = \frac{\sqrt{\pi}}{2 \sqrt{a}} \exp \left( \frac{b^2}{4a} \right) \mbox{erf} \left( \frac{2ax+b}{2 \sqrt{a}} \right)$

So continuing

$Z_{\lambda,\alpha} = \left. 2 \dfrac{\sqrt{\pi}}{2 \sqrt{a}} \exp \left( \dfrac{b^2}{4a} \right) \mbox{erf} \left( \dfrac{2ax+b}{2 \sqrt{a}} \right) \right|^{0}_{-\infty}$

$ = \left. \dfrac{\sqrt{\pi}}{\sqrt{a}} \exp \left( \dfrac{b^2}{4a} \right) \left( 2\Phi \left( \sqrt{2} \dfrac{2ax+b}{2 \sqrt{a}} \right) - 1 \right) \right|^{0}_{-\infty}$

$ = \left. \dfrac{\sqrt{\pi}}{\sqrt{a}} \exp \left( \dfrac{b^2}{4a} \right) \left( 2\Phi \left( \dfrac{2ax+b}{ \sqrt{2a}} \right) - 1 \right) \right|^{0}_{-\infty}$

$ = \left. \dfrac{\sqrt{\pi}}{\sqrt{a}} \exp \left( \dfrac{b^2}{4a} \right) \left( 2\Phi \left( \dfrac{2ax+b}{ \sqrt{2a}} \right) - 1 \right) \right|^{0}_{-\infty}$

$ = 2 \dfrac{\sqrt{\pi}}{\sqrt{a}} \exp \left( \dfrac{b^2}{4a} \right) \Phi \left( \dfrac{b}{ \sqrt{2a}} \right) $

$ = 2 \dfrac{\sqrt{\pi}}{\sqrt{\alpha \lambda}} \exp \left( \dfrac{\lambda^2(1-\alpha)^2}{4\lambda \alpha} \right) \Phi \left( \dfrac{\lambda(1-\alpha)}{ \sqrt{2\lambda \alpha}} \right) $

$ = 2 \dfrac{\sqrt{\pi}}{\sqrt{\alpha \lambda}} \exp \left( \dfrac{\lambda(1-\alpha)^2}{4 \alpha} \right) \Phi \left( \dfrac{\sqrt{\lambda}(1-\alpha)}{ \sqrt{2 \alpha}} \right) $

Am I going wrong somewhere?