For me, the clearest way to think about matrix differentiation is as a special case of differentiation on real or complex vector spaces. In this answer, I'll stick to real vector spaces, because the complex case might have subtleties I'm not aware of. The definition I'm going to present may not be equivalent to the standard definition (given by Harald Hanche-Olsen), but I think it's at least morally correct, and I find it very useful in practice. Corrections to make it more rigorous would be welcome!

[Edit: Harald Hanche-Olsen has let me know that if you drop the linearity and boundedness condition from my definition, you get the Gâteaux derivative. It follows that the derivative I've defined is not equivalent to the standard derivative, the Fréchet derivative, because a function can have a linear Gâteaux derivative even if the Fréchet derivative doesn't exist. If the Fréchet derivative does exist, however, the Gâteaux derivative also exists, and is equal to the Fréchet derivative.]

Without further ado, let $E$ be a real vector space, and let $F$ be a real Banach space. (In the first example you mentioned, $E$ would be the set of $n \times m$ matrices, and $F$ would be the reals. In your second example, $E$ would be the reals, and $F$ would be the $n \times m$ matrices with a suitable norm---maybe the uniform norm?) We want to define the derivative of a function $f \colon E \to F$. For each point $x \in E$, let $df_x \colon E \to F$ be the function

$$df_x(v) = \frac{d}{d\epsilon} f(x + \epsilon v)|_{\epsilon = 0},$$

where $\epsilon$ is a real number. If this function exists and is linear, it's called the derivative of $f$ at $x$. (Actually, if $F$ is infinite-dimensional, we also have to demand that $df_x$ be bounded. If $F$ is finite-dimensional, every linear map into $F$ is bounded, so we don't have to worry about it. See the footnote for more details.)

I like this definition because it spotlights the fact that $df_x(v)$ is the directional derivative of $f$ along $v$. If you want to work with partial derivatives, you can pick a basis $e_{1}, e_{2}, e_{3} \ldots$ for $E$ and define $\partial_i f(x) = df_x(e_i)$. It follows from the linearity condition that

$$df_x(\beta_{1} e_{1} + \ldots + \beta_{n} e_{n}) = \beta_{1} \partial_{1} f(x) + \ldots + \beta_{n} \partial_{n} f(x).$$

What about higher derivatives? Again, Harald Hanche-Olsen has given the standard definition, and that definition will work with any definition of the first derivative. However, I get a little squitchy thinking about nested function spaces like $B(E, B(E, B(E, F)))$. I wish there were a better way! Intuitively, you'd want the second derivative to be something like this:

$$d^{2} f_{x}(v_{1}, v_{2}) = \frac{d}{d\epsilon_{1}} \frac{d}{d\epsilon_{2}} f(x + \epsilon_{1} v_{1} + \epsilon_{2} v_{2})|{\epsilon{1} = \epsilon_{2} = 0},$$

with, of course, conditions about existence, bilinearity, and boundedness. (You might also have to explicitly demand symmetry in the arguments $v_{1}$ and $v_{2}$.) I would be very interested to know whether (or under what conditions) this definition is equivalent to the standard one! In fact, I might even ask a question about it...

-------

Footnote-------

To be thorough, I should say how the derivative

$$\frac{d}{d\epsilon} f(x + \epsilon v)|_{\epsilon = 0}$$

is defined! It's just the usual derivative from intro analysis:

$$\frac{d}{d\epsilon} f(x + \epsilon v) |{\epsilon = 0} = \lim{\epsilon \to 0} \frac{1}{\epsilon}[f(x + \epsilon v) - f(x)].$$

The notion of a limit is well-defined (although the limit is not guaranteed to exist) because $F$, being a Banach space, gets a topology from its norm.

p.s. Sorry about the LaTeX! I swear it's working in the preview.

3 Yet another attempt at LaTeX repair

For me, the clearest way to think about matrix differentiation is as a special case of differentiation on real or complex vector spaces. In this answer, I'll stick to real vector spaces, because the complex case might have subtleties I'm not aware of. The definition I'm going to present may not be equivalent to the standard definition (given by Harald Hanche-Olsen), but I think it's at least morally correct, and I find it very useful in practice. Corrections to make it more rigorous would be welcome!

Without further ado, let $E$ be a real vector space, and let $F$ be a real Banach space. (In the first example you mentioned, $E$ would be the set of $n \times m$ matrices, and $F$ would be the reals. In your second example, $E$ would be the reals, and $F$ would be the $n \times m$ matrices with a suitable norm---maybe the uniform norm?) We want to define the derivative of a function $f \colon E \to F$. For each point $x \in E$, let $df_x \colon E \to F$ be the function

$df_x(v)$df_x(v) = \frac{d}{d\epsilon} f(x + \epsilon v)|_{\epsilon = 0},$0},$$where \epsilon is a real number. If this function exists and is linear, it's called the derivative of f at x. (Actually, if F is infinite-dimensional, we also have to demand that df_x be bounded. If F is finite-dimensional, every linear map into F is bounded, so we don't have to worry about it. See the footnote for more details.) I like this definition because it spotlights the fact that df_x(v) is the directional derivative of f along v. If you want to work with partial derivatives, you can pick a basis e_{1}, e_{2}, e_{3} \ldots for E and define \partial_i f(x) = df_x(e_i). It follows from the linearity condition that df_x(\beta_{1} df_x(\beta_{1} e_{1} + \ldots + \beta_{n} e_{n}) = \beta_{1} \partial_{1} f(x) + \ldots + \beta_{n} \partial_{n} f(x).f(x).$$ What about higher derivatives? Again, Harald Hanche-Olsen has given the standard definition, and that definition will work with any definition of the first derivative. However, I get a little squitchy thinking about nested function spaces like$B(E, B(E, B(E, F)))$. I wish there were a better way! Intuitively, you'd want the second derivative to be something like this:$d^{2} f_x(v_{1}$d^{2} f_{x}(v_{1}, v_{2}) = \frac{d}{d\epsilon_{1}} \frac{d}{d\epsilon_{2}} f(x + \epsilon_{1} v_{1} + \epsilon_{2} v_{2})|{\epsilon{1} = \epsilon_{2} = 0},$0},$$with, of course, conditions about existence, bilinearity, and boundedness. (You might also have to explicitly demand symmetry in the arguments v_{1} and v_{2}.) I would be very interested to know whether (or under what conditions) this definition is equivalent to the standard one! In fact, I might even ask a question about it... -------Footnote------- To be thorough, I should say how the derivative \frac{d}{d\epsilon} \frac{d}{d\epsilon} f(x + \epsilon v)|_{\epsilon = 0}0}$$

is defined! It's just the usual derivative from intro analysis:

$\frac{d}{d\epsilon}$\frac{d}{d\epsilon} f(x + \epsilon v) |{\epsilon = 0} = \lim{\epsilon \to 0} \frac{1}{\epsilon}[f(x + \epsilon v) - f(x)]$.f(x)].$$The notion of a limit is well-defined (although the limit is not guaranteed to exist) because$F$, being a Banach space, gets a topology from its norm. p.s. Sorry about the LaTeX! I swear it's working in the preview. 2 Seriously, this LaTeX engine is not working. For me, the clearest way to think about matrix differentiation is as a special case of differentiation on real or complex vector spaces. In this answer, I'll stick to real vector spaces, because the complex case might have subtleties I'm not aware of. The definition I'm going to present may not be equivalent to the standard definition (given by Harald Hanche-Olsen), but I think it's at least morally correct, and I find it very useful in practice. Corrections to make it more rigorous would be welcome! Without further ado, let$E$be a real vector space, and let$F$be a real Banach space. (In the first example you mentioned,$E$would be the set of$n \times m$matrices, and$F$would be the reals. In your second example,$E$would be the reals, and$F$would be the$n \times m$matrices with a suitable norm---maybe the uniform norm?) We want to define the derivative of a function$f \colon E \to F$. For each point$x \in E$, let$df_x \colon E \to F$be the function$df_x(v) = \left.\frac{d}{d\epsilon} frac{d}{d\epsilon} f(x + \epsilon v) \right|_{\epsilon v)|_{\epsilon = 0},$where$\epsilon$is a real number. If this function exists and is linear, it's called the derivative of$f$at$x$. (Actually, if$F$is infinite-dimensional, we also have to demand that$df_x$be bounded. If$F$is finite-dimensional, every linear map into$F$is bounded, so we don't have to worry about it. See the footnote for more details.) I like this definition because it spotlights the fact that$df_x(v)$is the directional derivative of$f$along$v$. If you want to work with partial derivatives, you can pick a basis$e_{1}, e_{2}, e_{3} \ldots$for$E$and define$\partial_i f(x) = df_x(e_i)$. It follows from the linearity condition that$df_x(\beta_{1} e_{1} + \ldots + \beta_{n} e_{n}) = \beta_{1} \partial_{1} f(x) + \ldots + \beta_{n} \partial_{n} f(x).$What about higher derivatives? Again, Harald Hanche-Olsen has given the standard definition, and that definition will work with any definition of the first derivative. However, I get a little squitchy thinking about nested function spaces like$B(E, B(E, B(E, F)))$. I wish there were a better way! Intuitively, you'd want the second derivative to be something like this:$d^{2}f_x(v_{1}, d^{2} f_x(v_{1}, v_{2}) = \left.\frac{d}{d\epsilon_{1}} frac{d}{d\epsilon_{1}} \frac{d}{d\epsilon_{2}} f(x + \epsilon_{1} v_{1} + \epsilon_{2} v_{2}) \right|v_{2})|{\epsilon{1} = \epsilon_{2} = 0},$with, of course, conditions about existence, bilinearity, and boundedness. (You might also have to explicitly demand symmetry in the arguments$v_{1}$and$v_{2}$.) I would be very interested to know whether (or under what conditions) this definition is equivalent to the standard one! In fact, I might even ask a question about it... -------Footnote------- To be thorough, I should say how the derivative$\left.\frac{d}{d\epsilon} \frac{d}{d\epsilon} f(x + \epsilon v) \right|_{\epsilon v)|_{\epsilon = 0}$is defined! It's just the usual derivative from intro analysis:$\left.\frac{d}{d\epsilon} \frac{d}{d\epsilon} f(x + \epsilon v) \right||{\epsilon = 0} = \lim{\epsilon \to 0} \frac{1}{\epsilon}[f(x + \epsilon v) - f(x)]$. The notion of a limit is well-defined (although the limit is not guaranteed to exist) because$F\$, being a Banach space, gets a topology from its norm.

p.s. Sorry about the LaTeX! I swear it's working in the preview.

1