I am wondering if there is any single step analysis for the Augmented Lagrangian method. Specifically, the problem is $$\min f(x) \text { s.t. } A x=b$$ where $f$ is convex, smooth. Such an objective with linear constraint may be the easiest to deal with. When we do the update, we solve $$\min _{x} f(x)+\overline{\lambda}^{T}(A x-b)+\frac{\rho}{2}\|A x-b\|^{2}=\mathcal{L}(x, \overline{\lambda} ; \rho)$$ by doing gradient update on this augmented Lagrangian and then update dual by $$\lambda=\overline{\lambda}+\rho(A x-b)$$

By saying "single step analysis", I want to bound the one-step progress $f\left(x^{k+1}\right)-f\left(x^{k}\right)$. Such a bound is easily derived for gradient descent, assuming that $f$ is continuous differentiable and has Lipschitz gradient $$\begin{aligned} f\left(x^{k+1}\right)-f\left(x^{k}\right) \leq\left\langle x^{k+1}-x^{k}, \nabla f\left(x^{k}\right)\right\rangle+\frac{L}{2}\left\|x^{k+1}-x^{k}\right\|^{2} \\=\left\langle-\alpha \nabla f\left(x^{k}\right),\right.& \nabla f\left(x^{k}\right) \rangle+\frac{L}{2} \alpha^{2}\left\|\nabla f\left(x^{k}\right)\right\|^{2} \\ &=-\alpha\left(1-\frac{L \alpha}{2}\right)\left\|\nabla f\left(x^{k}\right)\right\|^{2} \end{aligned}$$ This bound is in terms of $\left\|\nabla f\left(x^{k}\right)\right\|^{2} $. But I cannot find a similar one for Augmented Lagrangian method. You can make additional assumption, if necessary. Can anyone have a reference, or a way to derive that? Many thanks!


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.