I have just learned that a general framework in constrained optimization is called "proximal gradient optimization". It is interesting that the $\ell_0$ "norm" is also associated with a proximal operator. Hence, one can apply iterative hard thresholding algorithm to get the sparse solution of the following

$$\min \Vert Y-X\beta\Vert_F + \lambda \vert \beta \vert_0$$

If so, why people are still using $\ell_1$? If you can just get the result by non-convex optimization directly, why are people still using LASSO?

I want to know what's the downside of the proximal gradient approach for $\ell_0$ minimization. Is it because of the non-convexity and randomness associated with? That means the initial estimator is very important.

  • $\begingroup$ Using in what context? $\endgroup$ – Neal Oct 15 at 18:24
  • $\begingroup$ For example, at work, I don't really care what the scikit-learn or xgboost or statsmodels backends are doing as long as they fit a model correctly and in a reasonable amount of time. $\endgroup$ – Neal Oct 15 at 18:48
  • $\begingroup$ X-posted: math.stackexchange.com/q/3392929/339790 $\endgroup$ – Rodrigo de Azevedo Oct 15 at 20:10
  • $\begingroup$ @Neal, yes at work I don't care either. I am just curious.. $\endgroup$ – ArtificiallyIntelligence Oct 16 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.