Question on the relation between the Lagrangian Multiplier $\mathcal{L}=r+\lambda g(r,\theta_1,\dotsc,\theta_{N-1})$ and the Hessian of $r$

Question

I want to minimize the radius $r=\sqrt{x^2_1+x^2_2+\dotsb+x^2_N}$, with the constraint $g(r,\theta_1,\dotsc,\theta_{N-1})=0$. Here $g(r,\theta_1,\dotsc,\theta_{N-1})$ is a function defined in the $N$-sphere coordinate. This can be done by the Lagrange Multiplier method. The Lagrangian is constructed as follows $$ \mathcal{L}=r+\lambda g(r,\theta_1,\dotsc,\theta_{N-1}). $$ Then, a candidate for the minimal value of $r$ is obtained by solving the equations $\partial \mathcal{L}/\partial \lambda=g(r,\theta_1,\dotsc,\theta_{N-1})=0$ and $$ \frac{\partial\mathcal{L}}{\partial r}=1+\lambda \frac{\partial g}{\partial r}=0,\ \frac{\partial\mathcal{L}}{\partial \theta_j}=\lambda \frac{\partial g}{\partial \theta_j}=0,\ j=1,\dotsc,N-1. $$ Suppose I get a solution $(r_0,\theta_{1,0},\ldots,\theta_{N-1,0};\lambda_0)$ that satisfies the above equation. Now I can also solve the constraint $g(r,\theta_1,\dotsc,\theta_{N-1})=0$ which gives the implicit function $r=r(\theta_1,\theta_2,\dotsc,\theta_{N-1})$. I'm interested in the gradient and Hessian matrix of $r$ at $(\theta_{1,0},\dotsc,\theta_{N-1,0})$.

For the gradient, it is straightforward that $$ \frac{\partial r}{\partial \theta_j}\Bigr|_{\vec{\theta}=\vec{\theta}_0}=-\frac{\frac{\partial g}{\partial \theta_j}}{\frac{\partial g}{\partial r}}\Bigr|_{\vec{\theta}=\vec{\theta}_0,r=r_0}=0, $$ where I use the short notation $\vec{\theta}=(\theta_1,\dotsc,\theta_{N-1})$ and $\vec{\theta}_0=(\theta_{1,0},\dotsc,\theta_{N-1,0})$.

For the Hessian matrix defined by $$ H_{ij}=\frac{\partial^2 r}{\partial \theta_i \, \partial \theta_k} \Bigr|_{\vec{\theta}=\vec{\theta}_0}, $$ I wonder if the condition of $r_0$ being the minimal value impose any constraint on $H$, like what we encounter in the unconstrained case (where $H$ should be positive definite)?

Answer 1

If the constraint function $g$ allows to parametrize $r$ by $r = \varphi(\theta_1,\dots,\theta_{M})$ via the implicit function theorem, then you can rewrite your optimization problem as an unconstrained one $$\min_{\Theta \in \mathbb{R}^M} \varphi(\Theta),$$ because the constraint $g(\varphi(\Theta),\Theta) = 0$ is satisfied for every $\Theta = (\theta_1,\dots,\theta_M)$ by construction.

Since this is an unconstrained optimization problem, the first-order necessary optimality conditions in a minimum $\Theta$ for this problem would be $$0 = \nabla \varphi(\Theta) = - \bigl[\partial_r g(\varphi(\Theta),\Theta)\bigr]^{-1} \partial_\Theta g(\varphi(\Theta),\Theta).$$ (This is already is stated in OP.) Accordingly, the second-order necessary optimality condition (assuming that $g$ is twice continuously differentiable, because then so is $\varphi$ by implicit function black magic) would indeed be the positive semidefiniteness of the Hessian $$\nabla^2 \varphi(\Theta) \succeq 0.$$ Of course one can calculate the Hessian $\nabla^2 \varphi$ in a straightforward (but not very fun) manner by taking another derivative in the expression for $\nabla \varphi$ if one desires so. A sufficient optimality condition would be to have $\nabla^2 \varphi(\Theta) \succ 0$.

Addendum:

That being said, from the rudimentary problem description in OP, I would reasonably expect that the angle variables $\Theta$ will themselves be constrained to be between $0$ and $\frac\pi2$ or the likes. This would give a minimization problem with box constraints $$\min_{\Theta \in \mathbb R^M} \varphi(\Theta)\quad \text{subject to} \quad \Theta \in \Bigl\{ v \in \mathbb R^m \colon \ell_i \leq v_i \leq u_i\Bigr\}.$$ Calling the constraint set $B$ like box, the first-order optimality condition would become $$\langle \nabla \varphi(\Theta),\tau-\Theta\rangle \geq 0 \qquad \text{for all}~\tau \in B.$$ It is easy to see that this is equivalent to $$\partial_{\theta_i} \varphi(\Theta) \begin{cases} \geq 0 & \text{if}~\theta_i = \ell_i,\\ \leq 0 & \text{if}~\theta_i = u_i,\\ = 0 & \text{if}~\ell_i < \theta_i < u_i.\end{cases}$$ Further, the second-order necessary optimality condition would become $$\langle \nabla^2 \varphi(\Theta)\tau,\tau\rangle \geq 0 \qquad \text{for all}~\tau \in C(\Theta)$$ with the critical cone $$C(\Theta) = \Bigl\{v \in \mathbb{R}^M \colon \langle \nabla \varphi(\Theta),v\rangle = 0,~v_i \geq 0~\text{if}~\theta_i = \ell_i,~\text{and}~v_i \leq 0~\text{if}~\theta_i = u_i\Bigr\}.$$ So there is still a curvature condition for $\varphi(\Theta)$, but only along the critical cone $C(\Theta)$. Analogously, a sufficient condition would be $$\langle \nabla^2 \varphi(\Theta)\tau,\tau\rangle > 0 \qquad \text{for all}~\tau \in C(\Theta) \setminus \{0\}.$$

• You can find this in any textbook on Nonlinear Optimization. A standard reference could be the Nocedal/Wright book.
• Of course one could also a Lagrangian calculus for the box-constrained problem if one desires to do so.