Notions of Matrix Differentiation

Question

There are a few standard notions of matrix derivatives, e.g.

• If f is a function defined on the entries of a matrix A, then one can talk about the matrix of partial derivatives of f.
• If the entries of a matrix are all functions of a scalar x, then it makes sense to talk about the derivative of the matrix as the matrix of derivatives of the entries.

In the second case, it makes sense to talk about higher order derivatives, but in the first example the derivative provides a matrix from a scalar function, so you have to massage it a little bit to define a higher order derivative (e.g. take the trace of the resulting matrix).

I was wondering what other notions of matrix differentiation might exist out there, particularly any notions that allow for higher-order differentiation. I am also interested in any connections between various forms of matrix derivatives. As this is related to an undergraduate research project, I am mostly looking for answers that include a minimum of advanced terminology, but a discussion of how more general concepts (e.g. differential forms, matrix exponentials, etc.) relate to matrix derivatives would also be helpful.

Answer 1

There is another interpretation of Elisha's question that I think has not yet been addressed: How, and to what extent, can you do differential calculus with functional expressions of square matrices? For instance, how do you differentiate $\exp(A)$, which is defined for all square matrices $A$?

There is a good answer to this question for polynomials and an even better answer for the trace of a univariate polynomial. Both of these good answers extend automatically to analytic functions $f(z)$ evaluated at matrices by means of their Taylor series. (Which of course includes exponentiation.) I like to write the answer in terms of differentials. The differential of matrices $A$, $B$, etc., is $dA$, $dB$, etc., which you can take to mean that the matrices are unspecified functions of some dummy parameter $t$, and $dA$ then represents the formal numerator of the matrix-valued derivative $dA/dt$.

In this formalism, the Leibniz rule still holds, as long as you remember that matrix multiplication is non-commutative: $d(AB) = (dA)B + A(dB)$. Indeed, the matrices don't actually need to be square. The sum rule holds trivially. The negative power rule has a creative correct answer: $d(A^{-1}) = A^{-1}(dA)A^{-1}$. You can then differentiate exponentiation: $$d\exp(A) = dA + \frac{(dA)A + A(dA)}2 + \frac{(dA)A^2 + A(dA)A+A^2(dA)}6 + \cdots.$$ As you can see from this example, you can differentiate a power series, but nothing all that great happens because $dA$ might not commute with $A$. However, if you're computing the differential of $\mathrm{Tr}(f(A))$, then something very nice happens: You can cyclically permute each term of the differentiated power series to put $dA$ at the end. Thus, you get the very nice formula $$d\mathrm{Tr}(f(A)) = \mathrm{Tr}(f'(A)dA).$$ I have only derived this formula when $A$ is within the radius of convergence of the Taylor series of $f$. However, by continuity it applies generally when $f(A)$ and $f'(A)$ are well-defined. (For instance, if $A$ and $dA$ are both real symmetric or Hermitian, then it is enough for $f$ to exist as a real function and have one continuous derivative near the eigenvalues of $A$.)

Higher derivatives work basically the same way as first derivatives. Again, you should take $A$ to be a function of a dummy parameter $t$. To get the simplest expressions for higher derivatives, you should assume that $A$ is a linear function of $t$ (including a constant). Then for example: $$d^2\exp(A) = (dA)^2 + \frac{(dA)^2A + (dA)A(dA) + A(dA)^2}3 + \cdots.$$ The trace of this looks good at first, but the quadratic term in $A$ has both $\mathrm{Tr}((dA)^2A^2)$ and $\mathrm{Tr}((A(dA))^2)$, and I don't think that the rest of the traced Taylor series simplifies the way that it did for the first derivative.

A final remark: All of this works the best for functions $f(x)$ that are entire (in the sense of complex analysis, i.e., an infinite radius of convergence). One of the definitions of an entire function is one whose Taylor series decays superexponentially, and this is also a good condition for a non-commutative multivariate Taylor series, in the sense that it will converge for any matrices that you plug in.

Answer 2

In the first case, there is no difficulty in working with higher derivatives. All you have is a function $f$ of $n^2$ variables $a_{ij}$, and you can form and work with derivatives like $\partial^3f/\partial a_{12}\partial a_{23}\partial a_{34}$ with reckless abandon. Don't let the double indices worry you.

A more abstract approach would be to consider maps between any two finite dimensional spaces, or more generally Banach spaces. If $f\colon E\to F$ is a map between two such spaces and $x\in E$ then $f'(x)$, if it exists, is a bounded linear map, $f'(x)\in B(E,F)$, so that $f(x+h)=f(x)+f'(x)h+o(\|h\|)$ for $h\in E$ as $\|h\|\to0$. You want higher deriviatives? Well, $f'\colon E\to B(E,F)$ so we can employ the definition again and get a double derivative $f''(x)\in B(E,B(E,F))$. And so on. Actually, $B(E,B(E,F))$ is better identified with the space of bounded bilinear maps $E\times E\to F$, and the equality of mixed partial derivatives becomes the symmetry of $f''(x)$ in its two variables. You wanted to keep this simple, so I'll keep tensor products out of it for now.

(Quick addition: I forgot to mention that of course, the space of $n\times n$ matrices can stand in for either of the spaces $E$, $F$. Also, if you work with finite dimensional spaces only, you can delete the word “bounded” wherever it appears above.)

Answer 3

Since you say you are doing an undergraduate research project, I think the following document, The Matrix Cookbook, might be useful for you. There is a whole section devoted to computations of matrix derivatives. There is no deep mathematics going on, but it's a great reference.

Answer 4

Check out the chapter on matrix derivatives in "Some Eclectic Matrix Theory" by Kenneth S. Miller (1987, Robert E. Krieger Publishing Company). It has an elegant formulation that allows higher-order derivatives etc, and handles the case df(X)/dX where f(X) and X are both matrices by making the result also be a matrix.

Answer 5

For me, the clearest way to think about matrix differentiation is as a special case of differentiation on real or complex vector spaces. In this answer, I'll stick to real vector spaces, because the complex case might have subtleties I'm not aware of. The definition I'm going to present may not be equivalent to the standard definition (given by Harald Hanche-Olsen), but I think it's at least morally correct, and I find it very useful in practice. Corrections to make it more rigorous would be welcome!

[Edit: Harald Hanche-Olsen has let me know that if you drop the linearity and boundedness condition from my definition, you get the Gâteaux derivative. It follows that the derivative I've defined is not equivalent to the standard derivative, the Fréchet derivative, because a function can have a linear Gâteaux derivative even if the Fréchet derivative doesn't exist. If the Fréchet derivative does exist, however, the Gâteaux derivative also exists, and is equal to the Fréchet derivative.]

Without further ado, let $E$ be a real vector space, and let $F$ be a real Banach space. (In the first example you mentioned, $E$ would be the set of $n \times m$ matrices, and $F$ would be the reals. In your second example, $E$ would be the reals, and $F$ would be the $n \times m$ matrices with a suitable norm---maybe the uniform norm?) We want to define the derivative of a function $f \colon E \to F$. For each point $x \in E$, let $df_x \colon E \to F$ be the function

$$df_x(v) = \frac{d}{d\epsilon} f(x + \epsilon v)|_{\epsilon = 0},$$

where $\epsilon$ is a real number. If this function exists and is linear, it's called the derivative of $f$ at $x$. (Actually, if $F$ is infinite-dimensional, we also have to demand that $df_x$ be bounded. If $F$ is finite-dimensional, every linear map into $F$ is bounded, so we don't have to worry about it. See the footnote for more details.)

I like this definition because it spotlights the fact that $df_x(v)$ is the directional derivative of $f$ along $v$. If you want to work with partial derivatives, you can pick a basis $e_{1}, e_{2}, e_{3} \ldots$ for $E$ and define $\partial_i f(x) = df_x(e_i)$. It follows from the linearity condition that

$$df_x(\beta_{1} e_{1} + \ldots + \beta_{n} e_{n}) = \beta_{1} \partial_{1} f(x) + \ldots + \beta_{n} \partial_{n} f(x).$$

What about higher derivatives? Again, Harald Hanche-Olsen has given the standard definition, and that definition will work with any definition of the first derivative. However, I get a little squitchy thinking about nested function spaces like $B(E, B(E, B(E, F)))$. I wish there were a better way! Intuitively, you'd want the second derivative to be something like this:

$$d^{2} f_{x}(v_{1}, v_{2}) = \frac{d}{d\epsilon_{1}} \frac{d}{d\epsilon_{2}} f(x + \epsilon_{1} v_{1} + \epsilon_{2} v_{2})|_{\epsilon_{1} = \epsilon_{2} = 0},$$

with, of course, conditions about existence, bilinearity, and boundedness. (You might also have to explicitly demand symmetry in the arguments $v_{1}$ and $v_{2}$.) I would be very interested to know whether (or under what conditions) this definition is equivalent to the standard one! In fact, I might even ask a question about it...

---

Footnote

To be thorough, I should say how the derivative

$$\frac{d}{d\epsilon} f(x + \epsilon v)|_{\epsilon = 0}$$

is defined! It's just the usual derivative from intro analysis:

$$\frac{d}{d\epsilon} f(x + \epsilon v) |_{\epsilon = 0} = \lim_{\epsilon \to 0} \frac{1}{\epsilon}[f(x + \epsilon v) - f(x)].$$

The notion of a limit is well-defined (although the limit is not guaranteed to exist) because $F$, being a Banach space, gets a topology from its norm.

p.s. Sorry about the LaTeX! I swear it's working in the preview.