3
$\begingroup$

Question statement: Consider the space of probability measures with finite second moments $P_2(\mathbb{R}^d)$, which is equipped with the Wasserstein-2 distance $W_2$, and the square integrable function class $L_{2, \nu}(\mathbb{R}^D)$ with respect to a probability measure $\nu$. Let $S: P_2(\mathbb{R}^d) \rightarrow L_{2, \nu}(\mathbb{R}^D)$ be an operator such that $$(S\rho)(x) = \int \sigma(x; w) d\rho(w).$$ Here $\sigma$ is the "activation function", e.g., the sigmoid function. I am wondering if there are any sufficient conditions such that the operator $S$ is invertible and $S^{-1}$ is bounded.

My guess: For now I believe that $S$ is injective. My intuition comes from the universal function approximation theorem (UAT). Let $x = (x^0, \bar x)$ and $\sigma(x;w) = \sigma(w^T\bar x - x^0)$. Let $\rho_1,\rho_2 \in P_2(\mathbb{R}^d)$ such that $S\rho_1 = S\rho_2$. Assume that $$S\rho_1 - S\rho_2 = \int \sigma(\cdot; w) f(w) dw.$$ Then, by UAT, there exists a sequence of functions $f_n(w) = \sum_i b_i \sigma(w^T \bar x_i - x^0_i)$ such that $f_n \rightarrow f$ uniformly. Since $\int f_n f dw = 0$, it holds that $f = 0$ and $S\rho_1 - S\rho_2 = 0 \Rightarrow \rho_1 = \rho_2$. However, I am not sure if the above argument still holds true when $\rho$ does not have a density. Furthermore, I have no idea how to ensure that $S^{-1}$ is bounded.

$\endgroup$
3
  • $\begingroup$ Maybe I misunderstand something - but how can $S$ be surjective into $L_{2,\nu}(\mathbb{R}^D)$ when, actually, $S\rho \ge 0$ for each $\rho \in P_2(\mathbb{R}^d)$? $\endgroup$ Feb 19, 2021 at 12:02
  • 1
    $\begingroup$ @JochenGlueck is right, and the issue is worse than he states. You work like $\mathcal{P}_2(\mathbb{R}^D)$ were a vector space when it really is a convex subset of a (small) affine subspace of the vector space of finite signed Radon measure. Problem is wasserstein metric does not extend to a norm on that space (it behaves badly on affine segment). $\endgroup$ Jun 19, 2021 at 11:31
  • $\begingroup$ Perhaps this isn't your intention, but to have this line up for the usual single-hidden-layer neural networks, shouldn't we instead want $\sigma(x;w,b) = \sigma(w\cdot x + b)$ instead of what you described in your guess (which has one of the input neurons with a constant weight of -1)? $\endgroup$ Jul 21, 2021 at 4:09

2 Answers 2

1
$\begingroup$

This is just a partial answer regarding $S$ being injective.

The generalisation of your argument is given by Hornik (Theorem 5 and the definition of discriminatory functions above Theorem 5)

$\endgroup$
1
$\begingroup$

Here is a sufficient condition for non-injectivity under the assumption that $w=(\omega,\beta)$ and $\sigma(x;w)=\sigma(\omega\cdot x + \beta)$ i.e. an artificial neuron with activation $\sigma$, weights $\omega$, and bias $\beta$.

Claim: If $\sigma:\mathbb R\to\mathbb R$ is positive-homogenous (i.e. $\sigma(\lambda x)=\lambda \sigma(x)$ for all $\lambda\geq 0$ and $x\in\mathbb R$) then $S\delta_w=S(\tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0)$ for all $w$ and so $S$ is not injective.

Proof. To see why, note that positive-homogeneity implies $\sigma(x;2w)=2\sigma(x;w)$ and $\sigma(x;0)=0$ and $$ S[\delta_w](x) = \int \sigma(x;w')\,\mathrm d\delta_w(w') = \sigma(x;w) $$ Thus it follows that $$ S\left[\tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0\right](x) = \tfrac{1}{2}S[\delta_{2w}](x) + \tfrac{1}{2}S[\delta_0](x) = \tfrac{1}{2}\sigma(x;2w) + \tfrac{1}{2}\sigma(x;0) = \sigma(x;w) $$ Since $\delta_w\neq \tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0$ as measures (unless $w=0$) it follows that $S$ is not injective. $\blacksquare$

An example of such an activation function is the commonly used Rectified Linear Unit (ReLU) $\text{relu}(x)=\max(x,0)$. Thus merely appealing to the universal approximation theorem will not suffice in a search for an injectivity condition on $S$ as single-hidden layer neural networks with ReLU activations are universal.

In general, it should be possible to find obstructions to the injectivity of $S$ for other activation functions $\sigma$ whenever there are two single hidden-layer neural networks with activation $\sigma$ and linear output layer that are equal as functions (but not merely the result of a permutation of the hidden neurons and their connections).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.