3
$\begingroup$

The motivation for this question has to do with neural networks, but it is essentially a purely mathematical question.

Suppose you have a perceptron with one hidden layer, a bias, and a logistic activation function. That is a parametric function of the form

$$f(X) = \mathbf{B}\cdot\sigma\left(\mathbf{A}\mathbf{x}+\mathbf{b}\right)$$

where $\mathbf{x} \in \mathbb{R}^{n}$, $\mathbf{b} \in \mathbb{R}^{p}$, $\mathbf{A} \in \mathbb{R}^{n \times p}$, $\mathbf{B} \in \mathbb{R}^{p \times n}$, and $\sigma(x) = \frac{1}{1+e^{-x}}$ is the logistic function and is applied to vectors elementwise.

If $n = p$ and $\mathbf{A}$ and $\mathbf{B}$ are full rank, then $f$ is injective.

Question: if $p>n$, what conditions on $A$ and $B$ are sufficient for $f$ to be injective.

$\endgroup$
6
  • 1
    $\begingroup$ Why is this tagged fa.functional-analysis? What is the connection with functional analysis? $\endgroup$ Apr 22, 2014 at 17:59
  • $\begingroup$ This is really about transforming a structure by embedding it into a higher dimensional space and making sure you can get it back. If you still feel the tag is inappropriate, I'll remove it. $\endgroup$
    – Arthur B
    Apr 22, 2014 at 18:33
  • $\begingroup$ For that matter, this question appears to have nothing to do with differential topology. $\endgroup$ Apr 23, 2014 at 15:07
  • $\begingroup$ very well, I'm leaving linear algebra then $\endgroup$
    – Arthur B
    Apr 23, 2014 at 17:40
  • $\begingroup$ Did you ever find an answer to this? $\endgroup$
    – user76284
    Feb 8, 2022 at 8:32

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.