1
$\begingroup$

According to the wikipedia article: http://en.wikipedia.org/wiki/Levenberg_Marquardt

--

$S(\boldsymbol\beta+\boldsymbol\delta) \approx \|\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta\|^2$

Taking the derivative with respect to δ and setting the result to zero gives:

$(J^{T}J)\boldsymbol \delta = J^{T} [y - f(\boldsymbol \beta)])$

--

My attempt to derive the equation:

$\|\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta\|^2 = (\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)^T(\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)$

using product rule:

$\frac{\partial \|\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta\|^2}{\partial \boldsymbol\delta} = (-J^{T})(\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta) + (\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)^T(-J)$

The dimensions of the left and right side don't match. I believe there might be something wrong with my differentiation. There seems to be a transpose missing, but I'm not sure what would cause a transpose in the differentiation operation.

$\endgroup$
5
  • $\begingroup$ According to the FAQ this question seems to better suited for math.stackexchange.com. $\endgroup$
    – Dirk
    Commented Nov 16, 2010 at 5:24
  • $\begingroup$ The $J^T J$ operation on the Jacobian is needed since you are solving an overdetemined linear system in the least-squares sense, using the method of normal equations. (The additional diagonal term which distinguishes Levenberg-Marquardt from Gauss-Newton comes from an appropriate addition of rows on the original Jacobian). I would suggest looking first at the derivation of the normal equations and the Gauss-Newton method before considering the derivation of LM. $\endgroup$ Commented Nov 16, 2010 at 6:22
  • $\begingroup$ Perhaps some stuff listed at: en.wikipedia.org/wiki/Matrix_calculus may prove to be useful for you. Otherwise, yes, the question is more suited for math.SE $\endgroup$
    – Suvrit
    Commented Nov 16, 2010 at 8:56
  • $\begingroup$ This question has been asked again on math.SE, so I'm closing the question now. Perhaps someone can provide the link? $\endgroup$ Commented Nov 16, 2010 at 22:23
  • $\begingroup$ here: math.stackexchange.com/questions/10556/… $\endgroup$
    – Suvrit
    Commented Nov 17, 2010 at 9:04

1 Answer 1

2
$\begingroup$

This is very possibly a homework exercise, so I won't provide a complete solution. The computation of the gradient of $\| y-f(\beta)-J\delta \|^{2}$ simply isn't correct.

If $y$ is a vector of size $m$ by 1, $J$ is a matrix of size $m$ by $n$, and $\delta$ is a vector of size $n$ by 1,then

$(-J^{T})(\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)$

is of size $n$ by $1$, while

$(\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)^T(-J)$

is of size $1$ by $n$.

You don't have a correct version of the rule for computing the gradient of a dot product. Go back and review your vector calculus and try again.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged .