How to derive the Levenberg–Marquardt algorithm with matrix calculus [closed]

Question

According to the wikipedia article: http://en.wikipedia.org/wiki/Levenberg_Marquardt

--

$S(\boldsymbol\beta+\boldsymbol\delta) \approx \|\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta\|^2$

Taking the derivative with respect to δ and setting the result to zero gives:

$(J^{T}J)\boldsymbol \delta = J^{T} [y - f(\boldsymbol \beta)])$

--

My attempt to derive the equation:

$\|\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta\|^2 = (\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)^T(\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)$

using product rule:

$\frac{\partial \|\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta\|^2}{\partial \boldsymbol\delta} = (-J^{T})(\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta) + (\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)^T(-J)$

The dimensions of the left and right side don't match. I believe there might be something wrong with my differentiation. There seems to be a transpose missing, but I'm not sure what would cause a transpose in the differentiation operation.

Answer 1

This is very possibly a homework exercise, so I won't provide a complete solution. The computation of the gradient of $\| y-f(\beta)-J\delta \|^{2}$ simply isn't correct.

If $y$ is a vector of size $m$ by 1, $J$ is a matrix of size $m$ by $n$, and $\delta$ is a vector of size $n$ by 1,then

$(-J^{T})(\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)$

is of size $n$ by $1$, while

$(\mathbf{y} - \mathbf{f}(\boldsymbol\beta) - \mathbf{J}\boldsymbol\delta)^T(-J)$

is of size $1$ by $n$.

You don't have a correct version of the rule for computing the gradient of a dot product. Go back and review your vector calculus and try again.