MathOverflow will be down for maintenance for approximately 3 hours, starting Monday evening (06/24/2013) at approximately 9:00 PM Eastern time (UTC-4).
show/hide this revision's text 2 Small correction

There is another interpretation of Elisha's question that I think has not yet been addressed: How, and to what extent, can you do differential calculus with functional expressions of square matrices? For instance, how do you differentiate $\exp(A)$, which is defined for all square matrices $A$?

There is a good answer to this question for polynomials and an even better answer for the trace of a univariate polynomial. Both of these good answers extend automatically to analytic functions $f(z)$ evaluated at matrices by means of their Taylor series. (Which of course includes exponentiation.) I like to write the answer in terms of differentials. The differential of matrices $A$, $B$, etc., is $dA$, $dB$, etc., which you can take to mean that the matrices are unspecified functions of some dummy parameter $t$, and $dA$ then represents the formal numerator of the matrix-valued derivative $dA/dt$.

In this formalism, the Leibniz rule still holds, as long as you remember that matrix multiplication is non-commutative: $d(AB) = (dA)B + A(dB)$. Indeed, the matrices don't actually need to be square. The sum rule holds trivially. The inverse negative power rule has a creative correct answer: $d(A^{-1}) = A^{-1}(dA)A^{-1}$. You can then differentiate exponentiation: $$d\exp(A) = dA + \frac{(dA)A + A(dA)}2 + \frac{(dA)A^2 + A(dA)A+A^2(dA)}6 + \cdots.$$ As you can see from this example, you can differentiate a power series, but nothing all that great happens because $dA$ might not commute with $A$. However, if you're computing the differential of $\mathrm{Tr}(f(A))$, then something very nice happens: You can cyclically permute each term of the differentiated power series to put $dA$ at the end. Thus, you get the very nice formula $$d\mathrm{Tr}(f(A)) = \mathrm{Tr}(f'(A)dA).$$ I have only derived this formula when $A$ is within the radius of convergence of the Taylor series of $f$. However, by continuity it applies generally when $f(A)$ and $f'(A)$ are well-defined. (For instance, if $A$ and $dA$ are both real symmetric or Hermitian, then it is enough for $f$ to exist as a real function and have one continuous derivative near the eigenvalues of $A$.)

Higher derivatives work basically the same way as first derivatives. Again, you should take $A$ to be a function of a dummy parameter $t$. To get the simplest expressions for higher derivatives, you should assume that $A$ is a linear function of $t$ (including a constant). Then for example: $$d^2\exp(A) = (dA)^2 + \frac{(dA)^2A + (dA)A(dA) + A(dA)^2}3 + \cdots.$$ The trace of this looks good at first, but the quadratic term in $A$ has both $\mathrm{Tr}((dA)^2A^2)$ and $\mathrm{Tr}((A(dA))^2)$, and I don't think that the rest of the traced Taylor series simplifies the way that it did for the first derivative.

A final remark: All of this works the best for functions $f(x)$ that are entire (in the sense of complex analysis, i.e., an infinite radius of convergence). One of the definitions of an entire function is one whose Taylor series decays superexponentially, and this is also a good condition for a non-commutative multivariate Taylor series, in the sense that it will converge for any matrices that you plug in.
show/hide this revision's text 1

There is another interpretation of Elisha's question that I think has not yet been addressed: How, and to what extent, can you do differential calculus with functional expressions of square matrices? For instance, how do you differentiate $\exp(A)$, which is defined for all square matrices $A$?

There is a good answer to this question for polynomials and an even better answer for the trace of a univariate polynomial. Both of these good answers extend automatically to analytic functions $f(z)$ evaluated at matrices by means of their Taylor series. (Which of course includes exponentiation.) I like to write the answer in terms of differentials. The differential of matrices $A$, $B$, etc., is $dA$, $dB$, etc., which you can take to mean that the matrices are unspecified functions of some dummy parameter $t$, and $dA$ then represents the formal numerator of the matrix-valued derivative $dA/dt$.

In this formalism, the Leibniz rule still holds, as long as you remember that matrix multiplication is non-commutative: $d(AB) = (dA)B + A(dB)$. Indeed, the matrices don't actually need to be square. The sum rule holds trivially. The inverse rule has a creative correct answer: $d(A^{-1}) = A^{-1}(dA)A^{-1}$. You can then differentiate exponentiation: $$d\exp(A) = dA + \frac{(dA)A + A(dA)}2 + \frac{(dA)A^2 + A(dA)A+A^2(dA)}6 + \cdots.$$ As you can see from this example, you can differentiate a power series, but nothing all that great happens because $dA$ might not commute with $A$. However, if you're computing the differential of $\mathrm{Tr}(f(A))$, then something very nice happens: You can cyclically permute each term of the differentiated power series to put $dA$ at the end. Thus, you get the very nice formula $$d\mathrm{Tr}(f(A)) = \mathrm{Tr}(f'(A)dA).$$ I have only derived this formula when $A$ is within the radius of convergence of the Taylor series of $f$. However, by continuity it applies generally when $f(A)$ and $f'(A)$ are well-defined. (For instance, if $A$ and $dA$ are both real symmetric or Hermitian, then it is enough for $f$ to exist as a real function and have one continuous derivative near the eigenvalues of $A$.)

Higher derivatives work basically the same way as first derivatives. Again, you should take $A$ to be a function of a dummy parameter $t$. To get the simplest expressions for higher derivatives, you should assume that $A$ is a linear function of $t$ (including a constant). Then for example: $$d^2\exp(A) = (dA)^2 + \frac{(dA)^2A + (dA)A(dA) + A(dA)^2}3 + \cdots.$$ The trace of this looks good at first, but the quadratic term in $A$ has both $\mathrm{Tr}((dA)^2A^2)$ and $\mathrm{Tr}((A(dA))^2)$, and I don't think that the rest of the traced Taylor series simplifies the way that it did for the first derivative.

A final remark: All of this works the best for functions $f(x)$ that are entire (in the sense of complex analysis, i.e., an infinite radius of convergence). One of the definitions of an entire function is one whose Taylor series decays superexponentially, and this is also a good condition for a non-commutative multivariate Taylor series, in the sense that it will converge for any matrices that you plug in.