How should I understand the "$C^\infty$ functions" whose domain is the dual of $C^\infty(\mathbb{R}^n)$? I am reading Colombeau's book "New Generalized Functions and Multiplication of Distributions" and he uses the notation $C^\infty({C^\infty}'(\Omega))$ out of nowhere.
Here $\Omega$ is any open subset of some $\mathbb{R}^n$ and $C^\infty(\Omega)$ is the space of complex-valued $C^\infty$ functions on $\Omega$. ${C^\infty}'(\Omega)$ is the dual space of $C^\infty(\Omega)$.
So, my questions are:

*

*What topology is given on $C^\infty(\Omega)$ in general? Here $\Omega$ doesn't have to be bounded at all.. We need to deal with this issue in order to define ${C^\infty}'(\Omega)$.


*What topology is given on the dual space ${C^\infty}'(\Omega)$ then? This is necessary to define the notion of "$C^\infty$" for the functions whose domain is ${C^\infty}'(\Omega)$.
Could anyone clarify?
 A: The standard topology on $C^\infty(\Omega)$ is that of uniform convergence on compact subsets of $\Omega$ of all derivatives which is given by the seminorms $\|f\|_{K,n}=\sup\{|\partial^\alpha f(x)|: x\in K, |\alpha|\le n\}$ with $K\subseteq \Omega$ compact and $n\in \mathbb N$. This topology makes $C^\infty(\Omega)$ a nuclear Fréchet space. The standard topology on the dual is the topology of uniform convergence on all bounded (in this case, this is the same as on all compact) sets. This makes $C^\infty(\Omega)'=\mathscr E'(\Omega)$ (in Schwartz' notation, it is the space of distributions with compact support in $\Omega$) a very nice locally convex space (complete, ultrabornological, nuclear,...).
A: This is more like a long comment on the notion of smoothness than an actual answer, which has already been provided by Jochen Wengenroth. It tries to address the follow-up question the OP posted as a comment to that answer.
There are several, inequivalent ways to define smooth function(al)s over a (Hausdorff) locally convex vector space $X$ such as the space $\mathscr{E}'(\Omega)=C^\infty(\Omega)'$ of compactly supported distributions on a nonvoid open subset $\Omega\subset\mathbb{R}^n$ if one does not know anything else about $X$. This topic is discussed in depth by the book of H. H. Keller, Differential Calculus in Locally Convex Spaces (Springer Lecture Notes in Mathematics 417, Springer-Verlag, 1974), see e.g. Section 2.9, pp. 107-110 of said book. If $X$ is normable (e.g. Banach), then all such notions are equivalent. However, $X=\mathscr{E}'(\Omega)$ has no such property.
One way of defining such functions is the so-called Michal-Bastiani smoothness, which we will denote for now by $C^\infty_{\mathrm{MB}}$ (called $C^\infty_c$ in Keller's book - a poor choice of notation, in my opinion, since this is also used to denote spaces of smooth functions with compact support). One says that a function $F:X\rightarrow\mathbb{C}$ is Michal-Bastiani smooth or simply $C^\infty_{\mathrm{MB}}$ if its differentials of all
orders $k\in\mathbb{N}$ $$D^kF[\varphi](\vec{\varphi}_1,\ldots,\vec{\varphi}_k)=\left.\frac{\partial^k}{\partial t_1\cdots\partial t_k}\right|_{t_1=\cdots=t_k=0}F\left(\varphi+\sum^k_{j=1}t_j\vec{\varphi}_j\right)$$ define jointly continuous maps $$D^kF:X^{k+1}\ni(\varphi,\vec{\varphi}_1,\ldots,\vec{\varphi}_k)\mapsto D^kF[\varphi](\vec{\varphi}_1,\ldots,\vec{\varphi}_k)\in\mathbb{C}\ .$$ Another such way is by means of the notion of convenient smoothness, which is not discussed in Keller's book but gets quite a comprehensive treatment in the book of A. Kriegl and Peter W. Michor, The Convenient Setting of Global Analysis (American Mathematical Society, 1997). It starts from the observation that the notion of smooth curves valued in $X$ has no ambiguity at all and is defined just as in the case $X=\mathbb{R}^n$: a curve $\gamma:\mathbb{R}\rightarrow X$ is smooth if $$\gamma^{(k)}(t)=\frac{d^k}{dt^k}\gamma(t)$$ exists for all $k\in\mathbb{N}$ and all $t\in\mathbb{R}$, where the right hand side of the last formula above is defined recursively as in usual one-variable differential calculus. One then says that a map $F:X\rightarrow\mathbb{C}$ is (conveniently) smooth if it maps smooth ($X$-valued) curves to smooth ($\mathbb{C}$-valued) curves - more precisely, one has that $F\circ\gamma$ is a smooth $\mathbb{C}$-valued curve for all $X$-valued smooth curves $\gamma$ as above. This is, in fact, the notion of smoothness which is used in practice for calculus of variations.
Both notions of smoothness reduce to the usual notion if $X=\mathbb{R}^n$. This was already mentioned above for Michal-Bastiani smoothness as it is a special case of $X$ normable; as for convenient smoothness, this is a highly non-trivial result by J. Boman (Math. Scand. 20 (1967) 249-268, see also Theorem 3.4, pp. 26-27 of Kriegl-Michor). More generally, if $X$ is metrizable, then Michal-Bastiani smoothness and convenient smoothness are equivalent notions (see e.g. A. Frölicher, Smooth Structures, In: K. H. Kamps, D. Pumplün, W. Tholen, (eds.), Category Theory - Applications to Algebra, Logic and Topology, Springer Lecture Notes in Mathematics 962, Springer-Verlag, 1982, pp. 69-81, more precisely Theorem 1, pp. 77). Unfortunately, the locally convex (strong) topology ( = topology of uniform convergence on bounded subsets of $C^\infty(\Omega)$) on $X=\mathscr{E}'(\Omega)$ is not metrizable.
The actual notion of smoothness Colombeau employs in the work cited by the OP (J. F. Colombeau, New Generalized Functions and Multiplication of Distributions, North-Holland, 1984) is in fact defined elsewhere - more precisely, on the book Differential Calculus and Holomorphy - Real and Complex Analysis in Locally Convex Spaces by the same author (North-Holland, 1982). Namely, he uses the concept of Silva smoothness (named after the Portuguese mathematician José Sebastião e Silva), which is defined as follows.
Let $B\subset X$ be an absolutely convex bounded subset, i.e. a disk on $X$, and denote by $X_B$ the vector subspace generated by $B$, which is easily seen to be equal to $$X_B=\bigcup^\infty_{n=1}nB\ .$$ The Minkowski functional of $B$ $$\|\varphi\|_B=\inf\{\rho>0\ |\ \varphi\in\rho B\}$$ is finite for all $\varphi\in X_B$ and defines a norm $\|\cdot\|_B$ on $X_B$. It can be shown that if we endow $X_B$ with the norm topology induced by $\|\cdot\|_B$, then the natural inclusion map of $X_B$ into $X$ is continuous; moreover, if $B$ is in addition closed, then $B$ coincides with the closed unit ball of $X_B$ w.r.t. $\|\cdot\|_B$ (see e.g. Proposition 8.3.2, pp. 151 of the book by H. Jarchow, Locally Convex Spaces, B. G. Teubner, 1981 and the discussion that follows it). We say that a map $F:X\rightarrow\mathbb{C}$ is Silva smooth if for all disks $B$ on $X$ the restriction $F|_{X_B}$ is smooth in the usual sense for the normed vector spaces $X_B$. Recall now that on $X_B$ this notion of smoothness coincides with Michal-Bastiani smoothness - and, hence, with convenient smoothness - w.r.t. the topology induced by $\|\cdot\|_B$, which is finer than the topology induced by $X$ by the above result.
However, the notion of convenient smoothness on $X$ is unaltered if we replace the locally convex topology of $X$ by its bornologification, which is the finest locally convex topology on $X$ which renders the natural inclusion of $X_B$ into $X$ continuous for all disks $B$, which on its turn coincides with the finest locally convex topology on $X$ with the same bounded subsets as the original locally convex topology on $X$. This means that a curve $\gamma:\mathbb{R}\rightarrow X$ is smooth w.r.t. the original locally convex topology  of $X$ if and only if it is smooth w.r.t. the bornologification of $X$, whose restriction to $X_B$ coincides with the topology induced by $\|\cdot\|_B$ for all disks $B$ on $X$. Since it can be shown that the restriction of any smooth curve to a compact interval of $\mathbb{R}$ belongs to some disk $B$ on $X$, one concludes that Silva smoothness and convenient smoothness actually coincide for any (Hausdorff) locally convex vector space $X$. For details, see e.g. Corollary 1.8, pp. 13 of Kriegl-Michor and the discussion that precedes it. Which notion is more convenient (no pun intended), it depends on what you are trying to do.
In the case that $X=\mathscr{E}'(\Omega)$, we also have that $X$ is complete, hence it is locally complete (or convenient): for any disk $B$ on $X$, we have that a Cauchy sequence on $X_B$ w.r.t. $\|\cdot\|_B$ actually converges w.r.t. this norm (this is easy to see, since the topology induced on $X_B$ by $\|\cdot\|_B$ is finer that the one induced by $X$). In other words, any disk $B$ on $X$ is actually a so-called Banach disk. This is important because if $X$ is locally complete, then we can test smoothness of a curve $\gamma:\mathbb{R}\rightarrow X$ by composition with each $u\in X'$. More precisely, $\gamma$ is smooth if and only if $u\circ\gamma:\mathbb{R}\rightarrow\mathbb{C}$ is smooth for all $u\in X'$ (see e.g. Corollary 2.3, pp. 15 of Kriegl-Michor). Since in our case $X$ is also reflexive, we have that $X'=C^\infty(\Omega)$ in the sense that every $u\in X'$ is the evaluation at  $u\in C^\infty(\Omega)$. This provides a rather concrete picture of which are the smooth curves on $X=\mathscr{E}'(\Omega)$. Finally, in this case $X$ is also (resp. even ultra)bornological, meaning that its (strong) topology coincides with the latter's (resp. ultra)bornologification and therefore the topology induced by $\|\cdot\|_B$ on $X_B$ coincides with that induced by $X$ for all disks $B$ on $X$ (resp. which due to local completeness are also Banach disks). It could be that this implies that Silva smoothness (and hence convenient smoothness) also coincides with Michal-Bastiani smoothness for such an $X$, but I am not sure.
