# Optimization via Lagrange Multipliers [on hold]

Operating in a $\mathbb{R}^N$ Euclidean space, I have an objective function defined as follows:

$\phi(\vec{w}, b) = \frac{1}{2} \cdot \vec{w}^T \bullet \vec{w}$

In this particular case, I need to minimize this to get the optimal vector $\vec{w}$, and constant $b$, or more formally:

$\phi(\vec{w_o}, b_o) = \min\frac{1}{2} \cdot \vec{w}^T \bullet \vec{w}$

Next, the constraint imposed on finding the optimal solution are:

$\vec{w}^T \bullet \vec{x} + b + 1 = 0$

where $\vec{w} \in \mathbb{R}^N$, and $b \in \mathbb{R}$. The coordinate system $\vec{x}$ defines the points in this hyperspace, i.e., $x_1, x_2, \ldots, x_N$ - so for a 2-dimensional space, $\vec{x} = [x_1, x_2]$ (x, y axis).

So, that should qualify this problem for Lagrange Multipliers by definition - optimization of a convenient convex objective function and a linear constraint, so then I try that out:

$\Lambda(\vec{w}, b, \lambda) = \phi(\vec{w}, b) + \lambda(\vec{w}^T \bullet \vec{x_-} + b + 1)\\ = \frac{1}{2} \cdot \vec{w}^T \bullet \vec{w} + \lambda(\vec{w}^T \bullet \vec{x_-} + b + 1)$

And hence:

$\frac{\partial\Lambda}{\partial\vec{w}} = 0 = \vec{w} + \lambda \cdot \vec{x}$

$\frac{\partial\Lambda}{\partial b} = 0 = \lambda$

$\frac{\partial\Lambda}{\partial\lambda} = 0 = \vec{w}^T \bullet \vec{x} + b + 1$

Now what? What I should be able to do is to solve these 3 equations, but they don't seem to lead anywhere good.

To help place this in context, it may help to also post a link to the document which I'm writing, at the end of which I've hit this problem and hence posting here :)

-
Do you have a particular question in mind? –  Ryan Budney Feb 15 '10 at 8:17
If I'm understanding correctly, your problem is asking you to find the closest point on a hyperplane to the origin. Your 1st equation after "and hence" gives you the solution once you feed it into your constraint equation. No? –  Ryan Budney Feb 15 '10 at 9:01
Hi Ryan - Not quite... It's a problem similar in principal to the example on the Wikipedia page on Lagrange Multipliers - so I have a surface, and I need to optimize and find a point at which $\phi$ is minimized, subject to a constraint. The vector $\vec{w}$ and $b$ here are the parameters, and the vector $\vec{x}$ is the point in this Euclidean space $\mathbb{R}^N$ - where $N>1$. Please let me know if I've still failed to describe the problem and how, and I'll do my best - and thankyou for taking a look at it. –  user3989 Feb 15 '10 at 9:18
One point of confusion for me is that your function $\phi$ doesn't appear to depend on the variable x. –  Suresh Venkat Feb 15 '10 at 10:43
not to be rude, but the document you linked looks like a survey or a class project. Is this really an appropriate question for mathoverflow ? the question itself sounds like a standard SVM derivation as well. –  Suresh Venkat Feb 16 '10 at 0:38