How to derive the Levenberg–Marquardt algorithm with matrix calculus [closed]

According to the wikipedia article: http://en.wikipedia.org/wiki/Levenberg_Marquardt

--

$S(\boldsymbol\beta+\boldsymbol\delta) \approx \|\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta\|^2$

Taking the derivative with respect to δ and setting the result to zero gives:

$(J^{T}J)\boldsymbol \delta = J^{T} [y - f(\boldsymbol \beta)])$

--

My attempt to derive the equation:

$\|\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta\|^2 = (\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)^T(\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)$

using product rule:

$\frac{\partial \|\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta\|^2}{\partial \boldsymbol\delta} = (-J^{T})(\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta) + (\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)^T(-J)$

The dimensions of the left and right side don't match. I believe there might be something wrong with my differentiation. There seems to be a transpose missing, but I'm not sure what would cause a transpose in the differentiation operation.

-

closed as off topic by Scott Morrison♦Nov 16 '10 at 22:23

Questions on MathOverflow are expected to relate to research level mathematics within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here. If this question can be reworded to fit the rules in the help center, please edit the question.

According to the FAQ this question seems to better suited for math.stackexchange.com. –  Dirk Nov 16 '10 at 5:24
The $J^T J$ operation on the Jacobian is needed since you are solving an overdetemined linear system in the least-squares sense, using the method of normal equations. (The additional diagonal term which distinguishes Levenberg-Marquardt from Gauss-Newton comes from an appropriate addition of rows on the original Jacobian). I would suggest looking first at the derivation of the normal equations and the Gauss-Newton method before considering the derivation of LM. –  J. M. Nov 16 '10 at 6:22
Perhaps some stuff listed at: en.wikipedia.org/wiki/Matrix_calculus may prove to be useful for you. Otherwise, yes, the question is more suited for math.SE –  Suvrit Nov 16 '10 at 8:56
This question has been asked again on math.SE, so I'm closing the question now. Perhaps someone can provide the link? –  Scott Morrison Nov 16 '10 at 22:23
–  Suvrit Nov 17 '10 at 9:04

This is very possibly a homework exercise, so I won't provide a complete solution. The computation of the gradient of $\| y-f(\beta)-J\delta \|^{2}$ simply isn't correct.

If $y$ is a vector of size $m$ by 1, $J$ is a matrix of size $m$ by $n$, and $\delta$ is a vector of size $n$ by 1,then

$(-J^{T})(\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)$

is of size $n$ by $1$, while

$(\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)^T(-J)$

is of size $1$ by $n$.

You don't have a correct version of the rule for computing the gradient of a dot product. Go back and review your vector calculus and try again.

-