# Invertibility of neural network as operator on Wasserstein space

Question statement: Consider the space of probability measures with finite second moments $$P_2(\mathbb{R}^d)$$, which is equipped with the Wasserstein-2 distance $$W_2$$, and the square integrable function class $$L_{2, \nu}(\mathbb{R}^D)$$ with respect to a probability measure $$\nu$$. Let $$S: P_2(\mathbb{R}^d) \rightarrow L_{2, \nu}(\mathbb{R}^D)$$ be an operator such that $$(S\rho)(x) = \int \sigma(x; w) d\rho(w).$$ Here $$\sigma$$ is the "activation function", e.g., the sigmoid function. I am wondering if there are any sufficient conditions such that the operator $$S$$ is invertible and $$S^{-1}$$ is bounded.

My guess: For now I believe that $$S$$ is injective. My intuition comes from the universal function approximation theorem (UAT). Let $$x = (x^0, \bar x)$$ and $$\sigma(x;w) = \sigma(w^T\bar x - x^0)$$. Let $$\rho_1,\rho_2 \in P_2(\mathbb{R}^d)$$ such that $$S\rho_1 = S\rho_2$$. Assume that $$S\rho_1 - S\rho_2 = \int \sigma(\cdot; w) f(w) dw.$$ Then, by UAT, there exists a sequence of functions $$f_n(w) = \sum_i b_i \sigma(w^T \bar x_i - x^0_i)$$ such that $$f_n \rightarrow f$$ uniformly. Since $$\int f_n f dw = 0$$, it holds that $$f = 0$$ and $$S\rho_1 - S\rho_2 = 0 \Rightarrow \rho_1 = \rho_2$$. However, I am not sure if the above argument still holds true when $$\rho$$ does not have a density. Furthermore, I have no idea how to ensure that $$S^{-1}$$ is bounded.

• Maybe I misunderstand something - but how can $S$ be surjective into $L_{2,\nu}(\mathbb{R}^D)$ when, actually, $S\rho \ge 0$ for each $\rho \in P_2(\mathbb{R}^d)$? – Jochen Glueck Feb 19 at 12:02

## 1 Answer

This is just a partial answer regarding $$S$$ being injective.

The generalisation of your argument is given by Hornik (Theorem 5 and the definition of discriminatory functions above Theorem 5)