Functional derivatives on Banach spaces Physicists often use functional integrals and I'm trying to make sense of it in more precise terms. As you can see here, the functional derivative in Physics is defined in terms of Taylor expansions. Let me elaborate.
[The Physicist point of view] Let $f$ be a functional defined on some convenient function space and let $\varphi, \eta$ be fixed functions on this space. We expand $f$ in Taylor:
\begin{eqnarray}
f(\varphi + t\eta) = f(\varphi) + \frac{df}{dt}(\varphi + t\eta)\bigg{|}_{t=0}t + \frac{1}{2}\frac{d^{2}f}{dt^{2}}(\varphi+t\eta)\bigg{|}_{t=0}t^{2} +\cdots + \frac{1}{n!}\frac{d^{n}f}{dt^{n}}(\varphi+t\eta)\bigg{|}_{t=0}t^{n} +\mathcal{o}(t^{n+1}) \tag{1}\label{1}
\end{eqnarray}
where $\frac{d^{k}f}{dt^{k}}(\varphi+t\eta)\bigg{|}_{t=0}$ denotes the $k$-th Gâteaux derivative of $f$ at $\varphi$ evaluated at $\eta$. Thus, the $k$-th functional derivative of $f$ at $\eta$ is the function $\frac{\delta^{k} f}{\delta \varphi(x_{1})\cdots\delta\varphi(x_{k})}$ satisfying the equality:
\begin{eqnarray}
\frac{d^{k}f}{dt^{k}}(\varphi+t\eta)\bigg{|}_{t=0} =\int dx_{1}\cdots dx_{k} \frac{\delta^{k}f}{\delta \varphi(x_{1})\cdots\delta\varphi(x_{k})}\eta(x_{1})\cdots \eta(x_{k}) \tag{2}\label{2}
\end{eqnarray}
[The Mathematician point of view] Let $E, F$ be Banach spaces. A continuous bilinear functional ${\langle \cdot\,, \cdot \rangle }: E \times F \to \mathbb{R}$ is called $E$-non-degenerate if $\langle x,y\rangle = 0$ for all $y \in F$ implies $x=0$ (Similarly for $F$-non-degenerate). Equivalently, the two maps of $E$ to $F^{*}$ and $F$ to $E^{*}$ defined by $x \mapsto \langle x, \cdot \rangle$ and $y \mapsto \langle \cdot\,, y\rangle$, respectivelly, are one-to-one. If they are isomorphisms, $\langle \cdot\,, \cdot \rangle$ is called $E$ or $F$-strongly non-degenerate. We say that $E$ and $F$ are in duality if there is a non-degenerate bilinear functional $\langle \cdot\,, \cdot \rangle: E\times F \to \mathbb{R}$, also called a pairing of $E$ with $F$. If the functional is strongly non-degenerate, we say the duality is strong.
Consider the following definition (from this book).
Definition: Let $E$ and $F$ be normed spaces and $\langle \cdot, \cdot \rangle$ a $E$-non-degenerate pairing. Let $f: F \to \mathbb{R}$ be Fréchet differentiable at the point $\varphi \in F$ (denote this derivative as $Df(\varphi)$). The functional derivative $\delta f/\delta \varphi$ of $f$ with respect to $\varphi$ is the unique element in $E$, if it exists, such that:
\begin{eqnarray}
Df(\varphi)(\eta) = \left\langle \frac{\delta f}{\delta \varphi}, \eta\right\rangle\quad\forall\gamma \in F. \tag{3}\label{3}
\end{eqnarray}
Now, take $E=F=C(\Omega)$ to be a space of functions defined on a region $\Omega \subset \mathbb{R}^{n}$, which is Banach, and take the pairing $\langle \cdot, \cdot \rangle : C(\Omega)\times C(\Omega) \to \mathbb{R}$ given by:
\begin{eqnarray}
\langle f,g\rangle := \int_{\Omega}f(x)g(x)dx \tag{4}\label{4}
\end{eqnarray} 
If $f$ is Fréchet differentiable at $\varphi$, then it is also Gâteaux differentiable at $\varphi$ and the following identity holds:
\begin{eqnarray}
Df(\varphi)(\eta) = \frac{df}{dt}(\varphi+t\eta)\bigg{|}_{t=0} \tag{5}\label{5}
\end{eqnarray}
 Thus, the above definition together with the pairing \eqref{4} and \eqref{5} implies that the functional derivative of $f$ at $\varphi$ is the element $\delta f/\delta\varphi$ satisfying:
\begin{eqnarray}
\frac{df}{dt}(\varphi+t\eta)\bigg{|}_{t=0} = \int \frac{\delta f}{\delta \varphi}\eta(x)dx \tag{6}\label{6}
\end{eqnarray}
Note that \eqref{6} is exactly the physicist definition \eqref{2} for $k=1$. Now, my question is how to extend the mathematician's definition to consider higher order derivatives. If $f$ has, say, $k$ Fréchet derivatives at $\varphi$, then it has $k$ Gâteaux derivatives at this point. But now, the $k$-th Fréchet derivatives is a $k$-linear map, so I wonder if I should extend the definition by considering not pairings but $k$ linear maps instead, and then demand that these $k$-linear maps satisfy something like:
\begin{eqnarray}
D(\varphi_{1},\ldots,\varphi_{k})(\eta) = \left\langle \frac{\delta^{k}f}{\delta \varphi^{k}},\eta,\ldots,\eta\right\rangle \nonumber
\end{eqnarray}
where, now, $\langle \cdot, \cdots, \cdot \rangle$ is a $k$-linear non-degenerate map. Another possible approach is to use the same pairings and define high order derivatives as successive applications of the first derivative (I don't know how to do it though) and then prove a representation theorem when $E=F=C(\Omega)$, i.e. prove that if we take $E=F=C(\Omega)$ and use the pairing \eqref{4} then this $k$-th functional derivative becomes \eqref{2}. I'm really lost at this point, and I'd appreciate any help or tips on how to proceed. 
EDIT: A nice discussion in my previous question led me to some clarifications and possible directions. First, suppose that $f$ is twice Fréchet differentiable at $\varphi \in E$. Then, there exists a bounded bilinear functional $D^{2}f[\varphi]$ satisfying
\begin{eqnarray}
\lim_{\eta \to 0}\frac{Df[\varphi+\eta](\gamma)-Df[\varphi)](\gamma)-D^{2}f[\varphi](\eta,\gamma)}{\Vert\eta\Vert} = 0. \tag{7}\label{7}
\end{eqnarray}
But, using \eqref{3}, we also have 
$$
\begin{split}
Df[\varphi+\eta](\gamma)-Df[\varphi](\gamma) &= \left\langle \frac{\delta f}{\delta(\varphi+\eta)},\gamma\right\rangle - \left\langle\frac{\delta f}{\delta \varphi},\gamma\right\rangle \\
 &=\left\langle\frac{\delta f}{\delta(\varphi+\eta)}-\frac{\delta f}{\delta \varphi},\gamma\right\rangle = \langle \mathcal{L}[\varphi](\eta),\gamma\rangle
\end{split}
$$ for some linear operator $\mathcal{L}[\varphi]:E\mapsto E$. If we take $E=F=C(\Omega)$ as I mentioned before, it seems that the physicist's result is obtained by taking
$$
\begin{eqnarray}
\mathcal{L}[\varphi](\eta) := \int \frac{\delta^{2}f}{\delta \varphi^{2}} (x,y)\beta(x) dx \tag{8}\label{8}
\end{eqnarray}
$$
where, now, $\delta^{2}f/\delta\varphi^{2} = \delta^{2}f/\delta\varphi^{2}(x,y)$ is a function on $C(\Omega\times\Omega)$ and this would be our second order functional derivative of $f$. But I still have doubts about that. Why taking \eqref{8} as my linear map? It seems very arbitrary. 
 A: The `functional derivative' $\frac{\delta f}{\delta \varphi}$ in your sense is the gradient of the derivative $d f(\varphi)\in L(E,\mathbb R)$ with respect to duality $\langle\quad,\quad\rangle$ which you are specifying. It need not exist in $F$ since $F$ might be smaller then the dual of $E$. $\frac{\delta^2 f}{\delta \varphi^2}$ then would be a second order gradient with respect to an extension of $\langle\quad,\quad\rangle$, whose existence is also not sure, but $d^2f(\varphi)$ exist as a bounded bilinear map $E\times E\to \mathbb R$. In your example with $E=C(\Omega)$ the second derivative or Hessian is, in general, if it exists, a measure on $\omega\times \Omega$, and not a function.
See here for a concise setting of all this.
Added:
How to extend $\langle\quad,\quad\rangle$? Since $d^2f(\varphi): E\times E\to \mathbb R$ is symmetric bilinear bounded, it linearizes to the projective tensor product as $E\hat\otimes E\to \mathbb R$. So it lies in the dual $L(E,E')$ of the projective tensor product and is symmetric.
It might lie in a subspace of $L(E,E')$, for example in the subspace of compact operators, which is $E'\hat{\hat\otimes} E'$ (under the assumption of the approximation property, or in $F \hat{\hat\otimes}F$ (this would be one extension of $\langle\quad,\quad\rangle$), depending on the properties of the functional.
The easiest way for you would be to let $F=E'$ and to let $\langle\quad,\quad\rangle$ be just the duality, and to use full dual spaces all around. Of course you have symmetry.
A: Not a great idea to use normed spaces here. In physics, one typically deals with a functional on a space of smooth functions. Via polarization, the higher derivatives become symmetric continuous multilinear maps on that space of smooth functions.
Then via the Schwartz Kernel Theorem the latter become continuous linear maps, i.e., Schwartz distributions. In other words, what physicists write as
$$
\frac{\delta^k f}{\delta\phi(x_1)\cdots\delta\phi(x_k)}
$$
is a distributional kernel. For a reference that explores the corresponding theory, see "Properties of field functionals and characterization of local functionals" by Brouder, Dang, Laurent-Gengoux and Rejzner.
