Skip to main content
edited tags
Link
Arthur B
  • 1.9k
  • 1
  • 14
  • 18
Source Link
Arthur B
  • 1.9k
  • 1
  • 14
  • 18

Injectivity of a linear logistic transform

The motivation for this question has to do with neural networks, but it is essentially a purely mathematical question.

Suppose you have a perceptron with one hidden layer, a bias, and a logistic activation function. That is a parametric function of the form

$$f(X) = \mathbf{B}\cdot\sigma\left(\mathbf{A}\mathbf{x}+\mathbf{b}\right)$$

where $\mathbf{x} \in \mathbb{R}^{n}$, $\mathbf{b} \in \mathbb{R}^{p}$, $\mathbf{A} \in \mathbb{R}^{n \times p}$, $\mathbf{B} \in \mathbb{R}^{p \times n}$, and $\sigma(x) = \frac{1}{1+e^{-x}}$ is the logistic function and is applied to vectors elementwise.

If $n = p$ and $\mathbf{A}$ and $\mathbf{B}$ are full rank, then $f$ is injective.

Question: if $p>n$, what conditions on $A$ and $B$ are sufficient for $f$ to be injective.