Invertibility of neural network as operator on Wasserstein space

Question

Question statement: Consider the space of probability measures with finite second moments $P_2(\mathbb{R}^d)$, which is equipped with the Wasserstein-2 distance $W_2$, and the square integrable function class $L_{2, \nu}(\mathbb{R}^D)$ with respect to a probability measure $\nu$. Let $S: P_2(\mathbb{R}^d) \rightarrow L_{2, \nu}(\mathbb{R}^D)$ be an operator such that $$(S\rho)(x) = \int \sigma(x; w) d\rho(w).$$ Here $\sigma$ is the "activation function", e.g., the sigmoid function. I am wondering if there are any sufficient conditions such that the operator $S$ is invertible and $S^{-1}$ is bounded.

My guess: For now I believe that $S$ is injective. My intuition comes from the universal function approximation theorem (UAT). Let $x = (x^0, \bar x)$ and $\sigma(x;w) = \sigma(w^T\bar x - x^0)$. Let $\rho_1,\rho_2 \in P_2(\mathbb{R}^d)$ such that $S\rho_1 = S\rho_2$. Assume that $$S\rho_1 - S\rho_2 = \int \sigma(\cdot; w) f(w) dw.$$ Then, by UAT, there exists a sequence of functions $f_n(w) = \sum_i b_i \sigma(w^T \bar x_i - x^0_i)$ such that $f_n \rightarrow f$ uniformly. Since $\int f_n f dw = 0$, it holds that $f = 0$ and $S\rho_1 - S\rho_2 = 0 \Rightarrow \rho_1 = \rho_2$. However, I am not sure if the above argument still holds true when $\rho$ does not have a density. Furthermore, I have no idea how to ensure that $S^{-1}$ is bounded.

Answer 1

This is just a partial answer regarding $S$ being injective.

The generalisation of your argument is given by Hornik (Theorem 5 and the definition of discriminatory functions above Theorem 5)

Answer 2

Here is a sufficient condition for non-injectivity under the assumption that $w=(\omega,\beta)$ and $\sigma(x;w)=\sigma(\omega\cdot x + \beta)$ i.e. an artificial neuron with activation $\sigma$, weights $\omega$, and bias $\beta$.

Claim: If $\sigma:\mathbb R\to\mathbb R$ is positive-homogenous (i.e. $\sigma(\lambda x)=\lambda \sigma(x)$ for all $\lambda\geq 0$ and $x\in\mathbb R$) then $S\delta_w=S(\tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0)$ for all $w$ and so $S$ is not injective.

Proof. To see why, note that positive-homogeneity implies $\sigma(x;2w)=2\sigma(x;w)$ and $\sigma(x;0)=0$ and $$ S[\delta_w](x) = \int \sigma(x;w')\,\mathrm d\delta_w(w') = \sigma(x;w) $$ Thus it follows that $$ S\left[\tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0\right](x) = \tfrac{1}{2}S[\delta_{2w}](x) + \tfrac{1}{2}S[\delta_0](x) = \tfrac{1}{2}\sigma(x;2w) + \tfrac{1}{2}\sigma(x;0) = \sigma(x;w) $$ Since $\delta_w\neq \tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0$ as measures (unless $w=0$) it follows that $S$ is not injective. $\blacksquare$

An example of such an activation function is the commonly used Rectified Linear Unit (ReLU) $\text{relu}(x)=\max(x,0)$. Thus merely appealing to the universal approximation theorem will not suffice in a search for an injectivity condition on $S$ as single-hidden layer neural networks with ReLU activations are universal.

In general, it should be possible to find obstructions to the injectivity of $S$ for other activation functions $\sigma$ whenever there are two single hidden-layer neural networks with activation $\sigma$ and linear output layer that are equal as functions (but not merely the result of a permutation of the hidden neurons and their connections).