Functional derivatives on Banach spaces

Question

Physicists often use functional integrals and I'm trying to make sense of it in more precise terms. As you can see here, the functional derivative in Physics is defined in terms of Taylor expansions. Let me elaborate.

[The Physicist point of view] Let $f$ be a functional defined on some convenient function space and let $\varphi, \eta$ be fixed functions on this space. We expand $f$ in Taylor: \begin{eqnarray} f(\varphi + t\eta) = f(\varphi) + \frac{df}{dt}(\varphi + t\eta)\bigg{|}_{t=0}t + \frac{1}{2}\frac{d^{2}f}{dt^{2}}(\varphi+t\eta)\bigg{|}_{t=0}t^{2} +\cdots + \frac{1}{n!}\frac{d^{n}f}{dt^{n}}(\varphi+t\eta)\bigg{|}_{t=0}t^{n} +\mathcal{o}(t^{n+1}) \tag{1}\label{1} \end{eqnarray} where $\frac{d^{k}f}{dt^{k}}(\varphi+t\eta)\bigg{|}_{t=0}$ denotes the $k$-th Gâteaux derivative of $f$ at $\varphi$ evaluated at $\eta$. Thus, the $k$-th functional derivative of $f$ at $\eta$ is the function $\frac{\delta^{k} f}{\delta \varphi(x_{1})\cdots\delta\varphi(x_{k})}$ satisfying the equality: \begin{eqnarray} \frac{d^{k}f}{dt^{k}}(\varphi+t\eta)\bigg{|}_{t=0} =\int dx_{1}\cdots dx_{k} \frac{\delta^{k}f}{\delta \varphi(x_{1})\cdots\delta\varphi(x_{k})}\eta(x_{1})\cdots \eta(x_{k}) \tag{2}\label{2} \end{eqnarray}

[The Mathematician point of view] Let $E, F$ be Banach spaces. A continuous bilinear functional ${\langle \cdot\,, \cdot \rangle }: E \times F \to \mathbb{R}$ is called $E$-non-degenerate if $\langle x,y\rangle = 0$ for all $y \in F$ implies $x=0$ (Similarly for $F$-non-degenerate). Equivalently, the two maps of $E$ to $F^{*}$ and $F$ to $E^{*}$ defined by $x \mapsto \langle x, \cdot \rangle$ and $y \mapsto \langle \cdot\,, y\rangle$, respectivelly, are one-to-one. If they are isomorphisms, $\langle \cdot\,, \cdot \rangle$ is called $E$ or $F$-strongly non-degenerate. We say that $E$ and $F$ are in duality if there is a non-degenerate bilinear functional $\langle \cdot\,, \cdot \rangle: E\times F \to \mathbb{R}$, also called a pairing of $E$ with $F$. If the functional is strongly non-degenerate, we say the duality is strong.

Consider the following definition (from this book).

Definition: Let $E$ and $F$ be normed spaces and $\langle \cdot, \cdot \rangle$ a $E$-non-degenerate pairing. Let $f: F \to \mathbb{R}$ be Fréchet differentiable at the point $\varphi \in F$ (denote this derivative as $Df(\varphi)$). The functional derivative $\delta f/\delta \varphi$ of $f$ with respect to $\varphi$ is the unique element in $E$, if it exists, such that: \begin{eqnarray} Df(\varphi)(\eta) = \left\langle \frac{\delta f}{\delta \varphi}, \eta\right\rangle\quad\forall\gamma \in F. \tag{3}\label{3} \end{eqnarray}

Now, take $E=F=C(\Omega)$ to be a space of functions defined on a region $\Omega \subset \mathbb{R}^{n}$, which is Banach, and take the pairing $\langle \cdot, \cdot \rangle : C(\Omega)\times C(\Omega) \to \mathbb{R}$ given by: \begin{eqnarray} \langle f,g\rangle := \int_{\Omega}f(x)g(x)dx \tag{4}\label{4} \end{eqnarray} If $f$ is Fréchet differentiable at $\varphi$, then it is also Gâteaux differentiable at $\varphi$ and the following identity holds: \begin{eqnarray} Df(\varphi)(\eta) = \frac{df}{dt}(\varphi+t\eta)\bigg{|}_{t=0} \tag{5}\label{5} \end{eqnarray} Thus, the above definition together with the pairing \eqref{4} and \eqref{5} implies that the functional derivative of $f$ at $\varphi$ is the element $\delta f/\delta\varphi$ satisfying: \begin{eqnarray} \frac{df}{dt}(\varphi+t\eta)\bigg{|}_{t=0} = \int \frac{\delta f}{\delta \varphi}\eta(x)dx \tag{6}\label{6} \end{eqnarray} Note that \eqref{6} is exactly the physicist definition \eqref{2} for $k=1$. Now, my question is how to extend the mathematician's definition to consider higher order derivatives. If $f$ has, say, $k$ Fréchet derivatives at $\varphi$, then it has $k$ Gâteaux derivatives at this point. But now, the $k$-th Fréchet derivatives is a $k$-linear map, so I wonder if I should extend the definition by considering not pairings but $k$ linear maps instead, and then demand that these $k$-linear maps satisfy something like: \begin{eqnarray} D(\varphi_{1},\ldots,\varphi_{k})(\eta) = \left\langle \frac{\delta^{k}f}{\delta \varphi^{k}},\eta,\ldots,\eta\right\rangle \nonumber \end{eqnarray} where, now, $\langle \cdot, \cdots, \cdot \rangle$ is a $k$-linear non-degenerate map. Another possible approach is to use the same pairings and define high order derivatives as successive applications of the first derivative (I don't know how to do it though) and then prove a representation theorem when $E=F=C(\Omega)$, i.e. prove that if we take $E=F=C(\Omega)$ and use the pairing \eqref{4} then this $k$-th functional derivative becomes \eqref{2}. I'm really lost at this point, and I'd appreciate any help or tips on how to proceed.

EDIT: A nice discussion in my previous question led me to some clarifications and possible directions. First, suppose that $f$ is twice Fréchet differentiable at $\varphi \in E$. Then, there exists a bounded bilinear functional $D^{2}f[\varphi]$ satisfying \begin{eqnarray} \lim_{\eta \to 0}\frac{Df[\varphi+\eta](\gamma)-Df[\varphi)](\gamma)-D^{2}f[\varphi](\eta,\gamma)}{\Vert\eta\Vert} = 0. \tag{7}\label{7} \end{eqnarray} But, using \eqref{3}, we also have $$ \begin{split} Df[\varphi+\eta](\gamma)-Df[\varphi](\gamma) &= \left\langle \frac{\delta f}{\delta(\varphi+\eta)},\gamma\right\rangle - \left\langle\frac{\delta f}{\delta \varphi},\gamma\right\rangle \\ &=\left\langle\frac{\delta f}{\delta(\varphi+\eta)}-\frac{\delta f}{\delta \varphi},\gamma\right\rangle = \langle \mathcal{L}[\varphi](\eta),\gamma\rangle \end{split} $$ for some linear operator $\mathcal{L}[\varphi]:E\mapsto E$. If we take $E=F=C(\Omega)$ as I mentioned before, it seems that the physicist's result is obtained by taking $$ \begin{eqnarray} \mathcal{L}[\varphi](\eta) := \int \frac{\delta^{2}f}{\delta \varphi^{2}} (x,y)\beta(x) dx \tag{8}\label{8} \end{eqnarray} $$ where, now, $\delta^{2}f/\delta\varphi^{2} = \delta^{2}f/\delta\varphi^{2}(x,y)$ is a function on $C(\Omega\times\Omega)$ and this would be our second order functional derivative of $f$. But I still have doubts about that. Why taking \eqref{8} as my linear map? It seems very arbitrary.

Answer 1

Premise: almost (if not) all derivations below are kept at a formal level, i.e. (apart from the notes, almost) no discussion of the hypotheses needed to make the result rigorous are given. This is because the question asks to show a way to extend a particular notion of derivative of functionals in order to include differentiability of order higher than one, not when this way of doing things is rigorously justified.

The problem. When trying to figure out how a general notion applies to narrower scope, I think that the best (or perhaps the simpler) approach is to start from the very basic, primitive general concept which originates it. In this case, the originating concept is that of functional derivative: a map $f: F\to G$ is between two (real) Banach spaces $F$ and $G$ is Gâteaux differentiable if its functional derivative exists and is linear respect to the increment, i.e. $$ \bigg{[}\frac{\mathrm{d}}{\mathrm{d}\varepsilon}f[\varphi+\varepsilon \eta]\bigg{]}_{\varepsilon = 0} \triangleq D f[\varphi](\eta)\in\mathscr{L\!i\!n}(F, G) \quad\forall \varphi,\eta\in F\label{A}\tag{A} $$ where $\varepsilon$ is a real parameter and $\mathscr{L\!i\!n}(F, G)$ is the vector space of all linear maps from $G$ to $F$ (including unbounded, and thus possibly discontinuous, ones). If the functional derivative is also continuous respect to the topology of $G$ i.e. it belongs to $\mathscr{L}(F, G)\subsetneq\mathscr{L\!i\!n}(F, G)$, then it is a Fréchet derivative, thus $f$ is said to be Fréchet differentiable.
Let's define a (possibly multilinear) higher order functional derivative analogous to \eqref{A}: the classical idea seems to have been first explicitly stated by Fantappiè in [1], §5, pp. 513-514 (see also [2], chapter VI, §(25÷27), pp. 70-78). If $\varepsilon_1,\ldots\varepsilon_k$ are $k\in\Bbb N\setminus\{0\}$ real parameters, for all $\varphi,\eta_i,\ldots,\eta_k \in F$, the $k$-th order functional derivative is recursively defined as follows: $$ \begin{split} D f[\varphi](\eta_1)&=\bigg{[}\frac{\partial}{\partial\varepsilon_1}f[\varphi+\varepsilon_1 \eta_1]\bigg{]}_{\varepsilon_1 = 0}\\ D^2 f[\varphi](\eta_1,\eta_2)&=\bigg{[}\frac{\partial^2}{\partial\varepsilon_2\partial\varepsilon_1}f[\varphi+\varepsilon_1 \eta_1+\varepsilon_2 \eta_2]\bigg{]}_{\varepsilon_1, \varepsilon_2= 0} \\ &\qquad=\bigg{[}\frac{\partial}{\partial\varepsilon_2}Df[\varphi+\varepsilon_2 \eta_2](\eta_1)\bigg{]}_{\varepsilon_2 = 0}\\ \vdots\;\qquad &\:\,\vdots\quad\quad\qquad\qquad\vdots\\ D^k f[\varphi](\eta_1,\ldots,\eta_k) &=\left[\frac{\partial^k}{\partial\varepsilon_k\cdots \partial\varepsilon_k}f\bigg[\varphi+\sum_{i=1}^k\varepsilon_i\eta_i\bigg]\right]_{\varepsilon_1,\ldots,\varepsilon_k= 0}\\ &\qquad=\bigg{[}\frac{\partial}{\partial\varepsilon_k}D^{n-1}f[\varphi+\varepsilon_k \eta_k](\eta_1,\ldots,\eta_{k-1}) \bigg{]}_{\varepsilon_k= 0} \end{split}\label{B}\tag{B} $$ Be it noted that $D^kf[\phi](\eta_1,\ldots,\eta_k)$, $k\ge1$ is a $k$-linear functional (Fantappiè, working with analytic functionals i.e. $ F=\mathscr{O}(\Bbb C)$ and $G=\Bbb C$ is also able to prove \eqref{A}, i.e. he does not need to assume that the functional derivative is linear, since the structure of functionals guarantees this fact, as shown in [1], §2, pp. 510-511 or [2], §25, pp. 73-74). Now applying the hierarchy \eqref{B} to equation \eqref{3} in the definition given by Abraham, Marsden and Ratiu, for all $\varphi,\eta_1,\ldots,\eta_k \in F$ we have $$ \begin{split} D^2 f[\varphi](\eta_1,\eta_2)&=\left[\frac{\partial}{\partial\varepsilon_2}\left\langle \frac{\delta f}{\delta \varphi}[\varphi+\varepsilon_2\eta_2], \eta_1\right\rangle\right]_{\varepsilon_2=0}\\ &\qquad= \left[\left\langle \frac{\partial}{\partial\varepsilon_2}\frac{\delta f}{\delta \varphi}[\varphi+\varepsilon_2\eta_2], \eta_1\right\rangle\right]_{\varepsilon_2=0}\\ &\qquad= \left\langle \frac{\delta^2 f}{\delta \varphi^2}[\varphi](\eta_2), \eta_1\right\rangle\simeq \left\langle \frac{\delta^2 f}{\delta \varphi^2}[\varphi], \eta_1, \eta_2\right\rangle_{\!\!1+2}\\ \vdots\;\qquad &\:\,\vdots\quad\quad\qquad\qquad\vdots\\ D^k f[\varphi](\eta_1,\ldots,\eta_k)&=\left[\frac{\partial}{\partial\varepsilon_k}\left\langle \frac{\delta^{k-1} f}{\delta \varphi^{k-1}}[\varphi+\varepsilon_k\eta_k], \eta_1,\ldots,\eta_{k-1} \right\rangle_{\!\!1+(k-1)}\right]_{\varepsilon_k=0}\\ &\qquad = \left\langle \frac{\delta^k f}{\delta \varphi^k}[\varphi](\eta_k), \eta_1,\ldots, \eta_{k-1}\right\rangle_{\!\!1+(k-1)} \!\!\simeq \left\langle \frac{\delta^k f}{\delta \varphi^k}[\varphi], \eta_1,\ldots, \eta_k\right\rangle_{\!\!1+k} \end{split}\label{C}\tag{C} $$ where $\langle{ \cdot\,, \ldots,\, \cdot}\rangle_{1+k}:E\times\big(\times^k_{i=1}F\big)\to\Bbb R$ is a $1+k$-linear mapping for all $k\ge 1$: thus if this $(1+k)$-linear mapping is definable, the $k$-th order functional derivative at $\varphi\in F$ of a functional $f: F\to\Bbb R$, $k\ge 1$ is the unique element $\delta^k f/\delta\varphi^k\in E$, if it exists, verifying the last of the equalities \eqref{C}.

Notes

• As stressed in the premise, the development above is by no means a proof of the existence (and uniqueness) of a $(1+k)$-linear mapping and an element in $E$ such that \eqref{C} holds. However, assuming that the hypotheses guarantee the existence of such objects, the formal step at the right side of each equality in \eqref{C} is justifiable in the same standard way used to prove isomorphisms theorems of the following kind $$ \mathscr{L}\big(F, \mathscr{L}(F,\Bbb R)\big)\simeq \mathscr{L}\big(\times_{i=1}^{2}F,\Bbb R\big)\triangleq \mathscr{L}_2(F,\Bbb R) $$
• Earlier work. The key information in the formalism expressed by \eqref{B} is that it emphasizes the intrinsic multilinearity of higher order functional derivatives. In general function spaces, the concept of product of functions may not be defined therefore it couldn't be possible to define polynomials: therefore the straight notation $$ \bigg{[}\frac{\mathrm{d}^k}{\mathrm{d}\varepsilon^k}f[\varphi+\varepsilon \eta]\bigg{]}_{\varepsilon = 0} \triangleq D^k f[\varphi](\eta^k) $$ could be misleading and overshadowing instead of shed light on the problems at hand. Nevertheless it is possible define abstract multilinear mappings and work with them. And while I have not being able to find earlier references for the notation \eqref{B} respect to the works of Fantappiè, Volterra is surely aware of the necessity of working in a multilinear setting: his development of higher order functional derivative (remember that he works essentially in a Banach space setting) leads exactly to formula \eqref{2} ([3], §2.7, p. 102, formula (5) or [3], §29, p. 25) and to an extension of Taylor's theorem but his approach, while being essentially the same is much more involved, and requires several restriction on the functional $f$ ([2] §2.5, p.99, or [3], §27, pp. 22-24).
• Life outside Banach spaces. As I briefly said in the main body of the question, formulas \eqref{B} were developed by Fantappiè to work with locally analytic functionals i.e. functionals defined on germs of holomorphic functions $ F=\mathscr{O}(\Bbb C)$ and $G=\Bbb C$. However, a more general and modern approach is developed by prof. Michor and described in the monograph linked in his answer: his approach is substantially based on the use of smooth/analytic curves with values in bornological spaces, i.e. on the use of functions of one real variable $c:\Bbb R\to B$ with $c\in C^\infty(\Bbb R, B)$ or $c\in C^\omega(\Bbb R, B)$, used as arguments of the functionals to be analyzed. This is truly an evolution of the classical methods by Volterra and Fantappiè, as it allows to give a precise meaning to straightforward expressions like the following one $$ \bigg{[}\frac{\mathrm{d}^k}{\mathrm{d}t^k}f[c(t)]\bigg{]}_{t = 0} \triangleq D^k f[c_0](c_1,\ldots,c_n)\quad c_0,c_1,\ldots,c_k\in B $$ for very general functionals.

References

[1] Fantappiè, Luigi, "La derivazione delle funzionali analitiche [Derivation of analytic functionals]" (Italian), Atti della Accademia Nazionale dei Lincei, Rendiconti, VI Serie, vol. 1, 1° semestre, pp. 509-514 (1925), JFM 51.0314.03.

[2] Fantappiè, Luigi, Teoría de los funcionales analíticos y sus aplicaciones. Curso de conferencias desarrollado en el Instituto de Matemáticas ""Jorge Juan"" de Madrid y en el Seminario Matemático de Barcelona en el año académico 1942-1943, recopiladas por R. Rodríguez Vidal (Catalan), Barcelona: Seminario Matemático de Barcelona [Imprenta-Escuela de la Casa Provincial de Caridad], pp. 174 (1943), MR0014598.

[3] Volterra, Vito, "Sulle funzioni che dipendono da altre funzioni [On functions which depend on other functions]" (in Italian), Atti della Reale Accademia dei Lincei, Rendiconti (4) III, No. 2, 97-105, 141-146, 153-158 (1887), JFM 19.0408.01.

[4] Volterra, Vito, Theory of functionals and of integral and integro-differential equations. Dover edition with a preface by Griffith C. Evans, a biography of Vito Volterra and a bibliography of his published works by Sir Edmund Whittaker. Unabridged republ. of the first English transl, New York: Dover Publications, Inc. pp. 39+XVI+226 (1959), MR0100765, Zbl 0086.10402.

Answer 2

The `functional derivative' $\frac{\delta f}{\delta \varphi}$ in your sense is the gradient of the derivative $d f(\varphi)\in L(E,\mathbb R)$ with respect to duality $\langle\quad,\quad\rangle$ which you are specifying. It need not exist in $F$ since $F$ might be smaller then the dual of $E$. $\frac{\delta^2 f}{\delta \varphi^2}$ then would be a second order gradient with respect to an extension of $\langle\quad,\quad\rangle$, whose existence is also not sure, but $d^2f(\varphi)$ exist as a bounded bilinear map $E\times E\to \mathbb R$. In your example with $E=C(\Omega)$ the second derivative or Hessian is, in general, if it exists, a measure on $\omega\times \Omega$, and not a function.

See here for a concise setting of all this.

Added:

How to extend $\langle\quad,\quad\rangle$? Since $d^2f(\varphi): E\times E\to \mathbb R$ is symmetric bilinear bounded, it linearizes to the projective tensor product as $E\hat\otimes E\to \mathbb R$. So it lies in the dual $L(E,E')$ of the projective tensor product and is symmetric. It might lie in a subspace of $L(E,E')$, for example in the subspace of compact operators, which is $E'\hat{\hat\otimes} E'$ (under the assumption of the approximation property, or in $F \hat{\hat\otimes}F$ (this would be one extension of $\langle\quad,\quad\rangle$), depending on the properties of the functional.

The easiest way for you would be to let $F=E'$ and to let $\langle\quad,\quad\rangle$ be just the duality, and to use full dual spaces all around. Of course you have symmetry.

Answer 3

Not a great idea to use normed spaces here. In physics, one typically deals with a functional on a space of smooth functions. Via polarization, the higher derivatives become symmetric continuous multilinear maps on that space of smooth functions. Then via the Schwartz Kernel Theorem the latter become continuous linear maps, i.e., Schwartz distributions. In other words, what physicists write as $$ \frac{\delta^k f}{\delta\phi(x_1)\cdots\delta\phi(x_k)} $$ is a distributional kernel. For a reference that explores the corresponding theory, see "Properties of field functionals and characterization of local functionals" by Brouder, Dang, Laurent-Gengoux and Rejzner.