MathOverflow is a question and answer site for professional mathematicians. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

According to the wikipedia article:


$S(\boldsymbol\beta+\boldsymbol\delta) \approx \|\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta\|^2$

Taking the derivative with respect to δ and setting the result to zero gives:

$(J^{T}J)\boldsymbol \delta = J^{T} [y - f(\boldsymbol \beta)])$


My attempt to derive the equation:

$\|\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta\|^2 = (\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)^T(\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)$

using product rule:

$\frac{\partial \|\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta\|^2}{\partial \boldsymbol\delta} = (-J^{T})(\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta) + (\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)^T(-J)$

The dimensions of the left and right side don't match. I believe there might be something wrong with my differentiation. There seems to be a transpose missing, but I'm not sure what would cause a transpose in the differentiation operation.

share|cite|improve this question

closed as off topic by Scott Morrison Nov 16 '10 at 22:23

Questions on MathOverflow are expected to relate to research level mathematics within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

According to the FAQ this question seems to better suited for – Dirk Nov 16 '10 at 5:24
The $J^T J$ operation on the Jacobian is needed since you are solving an overdetemined linear system in the least-squares sense, using the method of normal equations. (The additional diagonal term which distinguishes Levenberg-Marquardt from Gauss-Newton comes from an appropriate addition of rows on the original Jacobian). I would suggest looking first at the derivation of the normal equations and the Gauss-Newton method before considering the derivation of LM. – J. M. Nov 16 '10 at 6:22
Perhaps some stuff listed at: may prove to be useful for you. Otherwise, yes, the question is more suited for math.SE – Suvrit Nov 16 '10 at 8:56
This question has been asked again on math.SE, so I'm closing the question now. Perhaps someone can provide the link? – Scott Morrison Nov 16 '10 at 22:23
here:… – Suvrit Nov 17 '10 at 9:04

This is very possibly a homework exercise, so I won't provide a complete solution. The computation of the gradient of $\| y-f(\beta)-J\delta \|^{2}$ simply isn't correct.

If $y$ is a vector of size $m$ by 1, $J$ is a matrix of size $m$ by $n$, and $\delta$ is a vector of size $n$ by 1,then

$(-J^{T})(\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)$

is of size $n$ by $1$, while

$(\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)^T(-J)$

is of size $1$ by $n$.

You don't have a correct version of the rule for computing the gradient of a dot product. Go back and review your vector calculus and try again.

share|cite|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.