# Invertibility of neural network as operator on Wasserstein space

Question statement: Consider the space of probability measures with finite second moments $$P_2(\mathbb{R}^d)$$, which is equipped with the Wasserstein-2 distance $$W_2$$, and the square integrable function class $$L_{2, \nu}(\mathbb{R}^D)$$ with respect to a probability measure $$\nu$$. Let $$S: P_2(\mathbb{R}^d) \rightarrow L_{2, \nu}(\mathbb{R}^D)$$ be an operator such that $$(S\rho)(x) = \int \sigma(x; w) d\rho(w).$$ Here $$\sigma$$ is the "activation function", e.g., the sigmoid function. I am wondering if there are any sufficient conditions such that the operator $$S$$ is invertible and $$S^{-1}$$ is bounded.

My guess: For now I believe that $$S$$ is injective. My intuition comes from the universal function approximation theorem (UAT). Let $$x = (x^0, \bar x)$$ and $$\sigma(x;w) = \sigma(w^T\bar x - x^0)$$. Let $$\rho_1,\rho_2 \in P_2(\mathbb{R}^d)$$ such that $$S\rho_1 = S\rho_2$$. Assume that $$S\rho_1 - S\rho_2 = \int \sigma(\cdot; w) f(w) dw.$$ Then, by UAT, there exists a sequence of functions $$f_n(w) = \sum_i b_i \sigma(w^T \bar x_i - x^0_i)$$ such that $$f_n \rightarrow f$$ uniformly. Since $$\int f_n f dw = 0$$, it holds that $$f = 0$$ and $$S\rho_1 - S\rho_2 = 0 \Rightarrow \rho_1 = \rho_2$$. However, I am not sure if the above argument still holds true when $$\rho$$ does not have a density. Furthermore, I have no idea how to ensure that $$S^{-1}$$ is bounded.

• Maybe I misunderstand something - but how can $S$ be surjective into $L_{2,\nu}(\mathbb{R}^D)$ when, actually, $S\rho \ge 0$ for each $\rho \in P_2(\mathbb{R}^d)$? Commented Feb 19, 2021 at 12:02
• @JochenGlueck is right, and the issue is worse than he states. You work like $\mathcal{P}_2(\mathbb{R}^D)$ were a vector space when it really is a convex subset of a (small) affine subspace of the vector space of finite signed Radon measure. Problem is wasserstein metric does not extend to a norm on that space (it behaves badly on affine segment). Commented Jun 19, 2021 at 11:31
• Perhaps this isn't your intention, but to have this line up for the usual single-hidden-layer neural networks, shouldn't we instead want $\sigma(x;w,b) = \sigma(w\cdot x + b)$ instead of what you described in your guess (which has one of the input neurons with a constant weight of -1)? Commented Jul 21, 2021 at 4:09

This is just a partial answer regarding $$S$$ being injective.

The generalisation of your argument is given by Hornik (Theorem 5 and the definition of discriminatory functions above Theorem 5)

Here is a sufficient condition for non-injectivity under the assumption that $$w=(\omega,\beta)$$ and $$\sigma(x;w)=\sigma(\omega\cdot x + \beta)$$ i.e. an artificial neuron with activation $$\sigma$$, weights $$\omega$$, and bias $$\beta$$.

Claim: If $$\sigma:\mathbb R\to\mathbb R$$ is positive-homogenous (i.e. $$\sigma(\lambda x)=\lambda \sigma(x)$$ for all $$\lambda\geq 0$$ and $$x\in\mathbb R$$) then $$S\delta_w=S(\tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0)$$ for all $$w$$ and so $$S$$ is not injective.

Proof. To see why, note that positive-homogeneity implies $$\sigma(x;2w)=2\sigma(x;w)$$ and $$\sigma(x;0)=0$$ and $$S[\delta_w](x) = \int \sigma(x;w')\,\mathrm d\delta_w(w') = \sigma(x;w)$$ Thus it follows that $$S\left[\tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0\right](x) = \tfrac{1}{2}S[\delta_{2w}](x) + \tfrac{1}{2}S[\delta_0](x) = \tfrac{1}{2}\sigma(x;2w) + \tfrac{1}{2}\sigma(x;0) = \sigma(x;w)$$ Since $$\delta_w\neq \tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0$$ as measures (unless $$w=0$$) it follows that $$S$$ is not injective. $$\blacksquare$$

An example of such an activation function is the commonly used Rectified Linear Unit (ReLU) $$\text{relu}(x)=\max(x,0)$$. Thus merely appealing to the universal approximation theorem will not suffice in a search for an injectivity condition on $$S$$ as single-hidden layer neural networks with ReLU activations are universal.

In general, it should be possible to find obstructions to the injectivity of $$S$$ for other activation functions $$\sigma$$ whenever there are two single hidden-layer neural networks with activation $$\sigma$$ and linear output layer that are equal as functions (but not merely the result of a permutation of the hidden neurons and their connections).