Hi there, In my studies I come up with this nonconvex optimization problem argmin Ax_2+lamda*x_1 subject to x'x=1 where cost function is nonsmooth but convex and the constrant in nonconvex. I tries subgradient projection method for convex constraints but the global solution is not my desired solution. My question is that I should solve this problem hurestically or there is a reliable method for this nonconvex optimization problem?

You can have a look of these papers： 1. Jonathan H. Manton, Optimization algorithms exploiting unitary constraints. 2. Zaiwen Zai and Wotao Yin, A feasible method for optimization with orthogonality constraints. Wish these studies can help you. 


Conceptually, for algorithm design, the following version of the problem might be amenable to a larger number of techniques: \begin{equation*} \min_x\quad\Axb\^2\quad\text{s.t.}\quad \x\_1 \le \gamma,\quad\x\=1. \end{equation*} There are two reasons behind this reformulation:
Of course, several other numerical ideas also apply. For example, to get a good solution, you could start with $\gamma$ very large so that the $\ell_1$ constraint essentially disappears; then solve the problem exactly, and then gradually tighten $\gamma$. 

