How do I optimize over (or take derivative wrt) a square diagonal matrix?

2011-09-09T23:24:46Z

Hello. I'd like to solve the following optimization problem.

$P_i$ is a 6x6 matrix
$X$, $Y$ is a 6xk matrix
$w_i$ is a kx1 vector
$diag(w_i)$ is a square diagonal matrix with diagonal entries equal to $w_i$

$\min_{w_i} ~ ||P_i - X diag(w_i) Y^T||_F^2$

So the question is how to optimize over $diag(w_i)$.

Does anyone know how to take derivative wrt a diagonal matrix?

Or would it work if treat $diag(w_i)$ as a square matrix, solve it,
and then set off-diagonal entries to zeros?

Answer by Brian Borchers for How do I optimize over (or take derivative wrt) a square diagonal matrix?

2011-09-10T04:28:23Z

Your notation is somewhat confusing, in that you apply the subscript $i$ to $w$, and have a vector $w_{i}$, but don't use $i$ in any meaningful way in your problem. I'm going to take the liberty of rewriting the problem as

$\min_{w} \| P-X \mbox{diag}(w) Y^{T} \|_{F} $.

You may have a whole bunch of these problems to solve as $i$ varies over some index set, but each can be solved separately.

This is a linear least squares problem in disguise.

The key to seeing this is to recognize that the Frobenius norm of a matrix $Z$ is the two norm of the vector $\mbox{vec}(Z)$ obtained from the matrix $Z$ by stacking the columns of $Z$ one on top of another.

Also note that

$X \mbox{diag}(w) Y^{T}=\sum_{j=1}^{k} w_{j} X_{j}Y_{j}^{T}$

where $X_{j}$ is the $j$th column of $X$, and $Y_{j}$ is the $j$th column of $Y$.

Now, your problem can be written as

$\min_{w} \| P- \sum_{j=1}^{k} w_{j} X_{j}Y_{j}^{T} \|_{F}$.

Let $H_{j}=X_{j}Y_{j}^{T}$, for $j=1, 2, \ldots, k$. We now have

$\min_{w} \| P - \sum_{j=1}^{k} w_{j} H_{j} \|_{F}. $

Transforming this into vector form, this becomes

$\min_{w} \| \mbox{vec}(P) - \sum_{j=1}^{k} w_{j} \mbox{vec}(H_{j}) \|_{2}$.

Let $A$ be the matrix whose columns are given by

$A_{j}=\mbox{vec}(H_{j})$.

Then the optimization problem can be written as

$\min_{w} \| \mbox{vec}(P) - Aw \|_{2} $.

which is a conventional linear least squares problem.

How do I optimize over (or take derivative wrt) a square diagonal matrix? - MathOverflow

How do I optimize over (or take derivative wrt) a square diagonal matrix?

Answer by Brian Borchers for How do I optimize over (or take derivative wrt) a square diagonal matrix?