Skip to main content
MathOverflow

Return to Answer

Clarified a condition in the last paragraph.
Source Link
Christian Bueno
Christian Bueno
  • 459
  • 2
  • 7

Here is a sufficient condition for non-injectivity under the assumption that $w=(\omega,\beta)$ and $\sigma(x;w)=\sigma(\omega\cdot x + \beta)$ i.e. an artificial neuron with activation $\sigma$, weights $\omega$, and bias $\beta$.

Claim: If $\sigma:\mathbb R\to\mathbb R$ is positive-homogenous (i.e. $\sigma(\lambda x)=\lambda \sigma(x)$ for all $\lambda\geq 0$ and $x\in\mathbb R$) then $S\delta_w=S(\tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0)$ for all $w$ and so $S$ is not injective.

Proof. To see why, note that positive-homogeneity implies $\sigma(x;2w)=2\sigma(x;w)$ and $\sigma(x;0)=0$ and $$ S[\delta_w](x) = \int \sigma(x;w')\,\mathrm d\delta_w(w') = \sigma(x;w) $$ Thus it follows that $$ S\left[\tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0\right](x) = \tfrac{1}{2}S[\delta_{2w}](x) + \tfrac{1}{2}S[\delta_0](x) = \tfrac{1}{2}\sigma(x;2w) + \tfrac{1}{2}\sigma(x;0) = \sigma(x;w) $$ Since $\delta_w\neq \tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0$ as measures (unless $w=0$) it follows that $S$ is not injective. $\blacksquare$

An example of such an activation function is the commonly used Rectified Linear Unit (ReLU) $\text{relu}(x)=\max(x,0)$. Thus merely appealing to the universal approximation theorem will not suffice in a search for an injectivity condition on $S$ as single-hidden layer neural networks with ReLU activations are universal.

In general, it should be possible to find obstructions to the injectivity of $S$ for other activation functions $\sigma$ whenever there are two single hidden-hidden layerlayer neural networks with activation $\sigma$ and linear output layer that are equal as functions (but not merely the result of a permutation of the hidden neurons and their connections).

Here is a sufficient condition for non-injectivity under the assumption that $w=(\omega,\beta)$ and $\sigma(x;w)=\sigma(\omega\cdot x + \beta)$ i.e. an artificial neuron with activation $\sigma$, weights $\omega$, and bias $\beta$.

Claim: If $\sigma:\mathbb R\to\mathbb R$ is positive-homogenous (i.e. $\sigma(\lambda x)=\lambda \sigma(x)$ for all $\lambda\geq 0$ and $x\in\mathbb R$) then $S\delta_w=S(\tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0)$ for all $w$ and so $S$ is not injective.

Proof. To see why, note that positive-homogeneity implies $\sigma(x;2w)=2\sigma(x;w)$ and $\sigma(x;0)=0$ and $$ S[\delta_w](x) = \int \sigma(x;w')\,\mathrm d\delta_w(w') = \sigma(x;w) $$ Thus it follows that $$ S\left[\tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0\right](x) = \tfrac{1}{2}S[\delta_{2w}](x) + \tfrac{1}{2}S[\delta_0](x) = \tfrac{1}{2}\sigma(x;2w) + \tfrac{1}{2}\sigma(x;0) = \sigma(x;w) $$ Since $\delta_w\neq \tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0$ as measures (unless $w=0$) it follows that $S$ is not injective. $\blacksquare$

An example of such an activation function is the commonly used Rectified Linear Unit (ReLU) $\text{relu}(x)=\max(x,0)$. Thus merely appealing to the universal approximation theorem will not suffice in a search for an injectivity condition on $S$ as single-hidden layer neural networks with ReLU activations are universal.

In general, it should be possible to find obstructions to the injectivity of $S$ for other activation functions $\sigma$ whenever there are two single-hidden layer neural networks with activation $\sigma$ and linear output layer that are equal as functions.

Here is a sufficient condition for non-injectivity under the assumption that $w=(\omega,\beta)$ and $\sigma(x;w)=\sigma(\omega\cdot x + \beta)$ i.e. an artificial neuron with activation $\sigma$, weights $\omega$, and bias $\beta$.

Claim: If $\sigma:\mathbb R\to\mathbb R$ is positive-homogenous (i.e. $\sigma(\lambda x)=\lambda \sigma(x)$ for all $\lambda\geq 0$ and $x\in\mathbb R$) then $S\delta_w=S(\tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0)$ for all $w$ and so $S$ is not injective.

Proof. To see why, note that positive-homogeneity implies $\sigma(x;2w)=2\sigma(x;w)$ and $\sigma(x;0)=0$ and $$ S[\delta_w](x) = \int \sigma(x;w')\,\mathrm d\delta_w(w') = \sigma(x;w) $$ Thus it follows that $$ S\left[\tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0\right](x) = \tfrac{1}{2}S[\delta_{2w}](x) + \tfrac{1}{2}S[\delta_0](x) = \tfrac{1}{2}\sigma(x;2w) + \tfrac{1}{2}\sigma(x;0) = \sigma(x;w) $$ Since $\delta_w\neq \tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0$ as measures (unless $w=0$) it follows that $S$ is not injective. $\blacksquare$

An example of such an activation function is the commonly used Rectified Linear Unit (ReLU) $\text{relu}(x)=\max(x,0)$. Thus merely appealing to the universal approximation theorem will not suffice in a search for an injectivity condition on $S$ as single-hidden layer neural networks with ReLU activations are universal.

In general, it should be possible to find obstructions to the injectivity of $S$ for other activation functions $\sigma$ whenever there are two single hidden-layer neural networks with activation $\sigma$ and linear output layer that are equal as functions (but not merely the result of a permutation of the hidden neurons and their connections).

added 261 characters in body
Source Link
Christian Bueno
Christian Bueno
  • 459
  • 2
  • 7

Here is a sufficient condition for non-injectivity under the assumption that $w=(\omega,\beta)$ and $\sigma(x;w)=\sigma(\omega\cdot x + \beta)$ i.e. an artificial neuron with activation $\sigma$, weights $\omega$, and bias $\beta$.

Claim: If $\sigma:\mathbb R\to\mathbb R$ is positive-homogenous (i.e. $\sigma(\lambda x)=\lambda \sigma(x)$ for all $\lambda\geq 0$ and $x\in\mathbb R$) then $S\delta_w=S(\tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0)$ for all $w$ and so $S$ is not injective.

Proof. To see why, note that positive-homogeneity implies $\sigma(x;2w)=2\sigma(x;w)$ and $\sigma(x;0)=0$ and $$ S[\delta_w](x) = \int \sigma(x;w')\,\mathrm d\delta_w(w') = \sigma(x;w) $$ Thus it follows that $$ S\left[\tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0\right](x) = \tfrac{1}{2}S[\delta_{2w}](x) + \tfrac{1}{2}S[\delta_0](x) = \tfrac{1}{2}\sigma(x;2w) + \tfrac{1}{2}\sigma(x;0) = \sigma(x;w) $$ Since $\delta_w\neq \tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0$ as measures (unless $w=0$) it follows that $S$ is not injective. $\blacksquare$

An example of such an activation function is the commonly used Rectified Linear Unit (ReLU) $\text{relu}(x)=\max(x,0)$. Thus merely appealing to the universal approximation theorem will not suffice in a search for an injectivity condition on $S$ as single-hidden layer neural networks with ReLU activations are universal.

In general, it should be possible to find obstructions to the injectivity of $S$ for other activation functions $\sigma$ whenever there are two single-hidden layer neural networks with activation $\sigma$ and linear output layer that are equal as functions.

Here is a sufficient condition for non-injectivity under the assumption that $w=(\omega,\beta)$ and $\sigma(x;w)=\sigma(\omega\cdot x + \beta)$ i.e. an artificial neuron with activation $\sigma$, weights $\omega$, and bias $\beta$.

Claim: If $\sigma:\mathbb R\to\mathbb R$ is positive-homogenous (i.e. $\sigma(\lambda x)=\lambda \sigma(x)$ for all $\lambda\geq 0$ and $x\in\mathbb R$) then $S\delta_w=S(\tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0)$ for all $w$ and so $S$ is not injective.

Proof. To see why, note that positive-homogeneity implies $\sigma(x;2w)=2\sigma(x;w)$ and $\sigma(x;0)=0$ and $$ S[\delta_w](x) = \int \sigma(x;w')\,\mathrm d\delta_w(w') = \sigma(x;w) $$ Thus it follows that $$ S\left[\tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0\right](x) = \tfrac{1}{2}S[\delta_{2w}](x) + \tfrac{1}{2}S[\delta_0](x) = \tfrac{1}{2}\sigma(x;2w) + \tfrac{1}{2}\sigma(x;0) = \sigma(x;w) $$ Since $\delta_w\neq \tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0$ as measures (unless $w=0$) it follows that $S$ is not injective. $\blacksquare$

An example of such an activation function is the commonly used Rectified Linear Unit (ReLU) $\text{relu}(x)=\max(x,0)$. Thus merely appealing to the universal approximation theorem will not suffice in a search for an injectivity condition on $S$ as single-hidden layer neural networks with ReLU activations are universal.

Here is a sufficient condition for non-injectivity under the assumption that $w=(\omega,\beta)$ and $\sigma(x;w)=\sigma(\omega\cdot x + \beta)$ i.e. an artificial neuron with activation $\sigma$, weights $\omega$, and bias $\beta$.

Claim: If $\sigma:\mathbb R\to\mathbb R$ is positive-homogenous (i.e. $\sigma(\lambda x)=\lambda \sigma(x)$ for all $\lambda\geq 0$ and $x\in\mathbb R$) then $S\delta_w=S(\tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0)$ for all $w$ and so $S$ is not injective.

Proof. To see why, note that positive-homogeneity implies $\sigma(x;2w)=2\sigma(x;w)$ and $\sigma(x;0)=0$ and $$ S[\delta_w](x) = \int \sigma(x;w')\,\mathrm d\delta_w(w') = \sigma(x;w) $$ Thus it follows that $$ S\left[\tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0\right](x) = \tfrac{1}{2}S[\delta_{2w}](x) + \tfrac{1}{2}S[\delta_0](x) = \tfrac{1}{2}\sigma(x;2w) + \tfrac{1}{2}\sigma(x;0) = \sigma(x;w) $$ Since $\delta_w\neq \tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0$ as measures (unless $w=0$) it follows that $S$ is not injective. $\blacksquare$

An example of such an activation function is the commonly used Rectified Linear Unit (ReLU) $\text{relu}(x)=\max(x,0)$. Thus merely appealing to the universal approximation theorem will not suffice in a search for an injectivity condition on $S$ as single-hidden layer neural networks with ReLU activations are universal.

In general, it should be possible to find obstructions to the injectivity of $S$ for other activation functions $\sigma$ whenever there are two single-hidden layer neural networks with activation $\sigma$ and linear output layer that are equal as functions.

Source Link
Christian Bueno
Christian Bueno
  • 459
  • 2
  • 7

Here is a sufficient condition for non-injectivity under the assumption that $w=(\omega,\beta)$ and $\sigma(x;w)=\sigma(\omega\cdot x + \beta)$ i.e. an artificial neuron with activation $\sigma$, weights $\omega$, and bias $\beta$.

Claim: If $\sigma:\mathbb R\to\mathbb R$ is positive-homogenous (i.e. $\sigma(\lambda x)=\lambda \sigma(x)$ for all $\lambda\geq 0$ and $x\in\mathbb R$) then $S\delta_w=S(\tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0)$ for all $w$ and so $S$ is not injective.

Proof. To see why, note that positive-homogeneity implies $\sigma(x;2w)=2\sigma(x;w)$ and $\sigma(x;0)=0$ and $$ S[\delta_w](x) = \int \sigma(x;w')\,\mathrm d\delta_w(w') = \sigma(x;w) $$ Thus it follows that $$ S\left[\tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0\right](x) = \tfrac{1}{2}S[\delta_{2w}](x) + \tfrac{1}{2}S[\delta_0](x) = \tfrac{1}{2}\sigma(x;2w) + \tfrac{1}{2}\sigma(x;0) = \sigma(x;w) $$ Since $\delta_w\neq \tfrac{1}{2}\delta_{2w}+\tfrac{1}{2}\delta_0$ as measures (unless $w=0$) it follows that $S$ is not injective. $\blacksquare$

An example of such an activation function is the commonly used Rectified Linear Unit (ReLU) $\text{relu}(x)=\max(x,0)$. Thus merely appealing to the universal approximation theorem will not suffice in a search for an injectivity condition on $S$ as single-hidden layer neural networks with ReLU activations are universal.