I'm not a professional geometer. Thanks in advance for your patience.

So, let $n$, $k$, $p_0,\ldots,p_{k}$ be positive integers. Let $X$ (resp. $Y$) be an $p_0$-by-$n$ (resp. an $p_{k}$-by-$n$) real matrix. Consider the algebraic set $$ \Omega := \{(W,\Lambda) \in \mathcal R \times \mathcal X_+ | F_i(W,\Lambda) = 0,\;\forall i=0,\ldots,k\}, $$ where

- $F_i(W,\Lambda):=U_{i+1}^T(V_0XX^T-YX^T)V_{i+1}^T + \Lambda_i \circ W_i$, a polynomial in the matrices $W_j$, and in the
**entries**of the matrices $W_j$ and $\Lambda_j$. - $U_i := \Pi_{j=k}^{i}W_i\in \mathbb R^{p_i \times p_k}$,
- $V_i := \Pi_{j=i}^{k}W_i \in \mathbb R^{p_k \times p_i}$,
- $\mathcal R := \mathbb R^{p_0 \times p_1} \times \ldots \times \mathbb R^{p_{k-1} \times p_{k}}$, and
- $\mathcal X_+ := \mathbb R_+^{p_0 \times p_1} \times \ldots \times \mathbb R_+^{p_{k-1} \times p_{k}}$

Finally, for any $\Lambda \in \mathcal X_+$, define $\Omega_\Lambda := \{W|(W,\Lambda) \in \Omega\}$.

# Note

In defining the auxiliary set $\mathcal R$, I've taken the base field to be the real numbers $\mathbb R$. It might be that a cleaner picture is obtained by looking at a larger base object (e.g the complex numbers $\mathbb C$, etc.). Please feel free to change this to suite your needs.

**Defintion.** A *dense* matrix is a matrix with only nonzero entries.

# Question

- What can be said about the
*geometric / topological structure*of $\Omega_\Lambda$ (resp. of $\Omega$) ?- How many isolated points are there ? How many dense isolated points ? (in either case, a good upper bound is ok :) )
- What are the connected pieces of this set ?
- etc.

- What can be said about the asymptotic
*geoemetric / topological structure*of $\Omega_\Lambda$ in the limit $\Lambda \rightarrow 0$ ?

# Motivation

There is a rapidly growing branch of statistics called Deep Learning (ok most statisticians might disown it, but let's take this for granted). The idea is to approximate "any" function $f(x)$ via a product of matrices $W_i$, punctuated with nonlinear so-called "activation functions" $\sigma_i$ (a typical choice are *Rectified Linear Units* or *ReLUs* for short, defined by $\sigma_i(x):=\max(x,0)$). succinctly, $f(x) \approx \sigma_{k+1}(W_{k+1}(\sigma_k(\ldots(\sigma_0(W_0 x))\ldots)$. The matrices $W_i$ are the parameters of the model, and a optimized for a given task (this is curve-fitting).

Implementations of this simple idea yield super-human performance on certain task, including: computer vision (object recognition, etc.) and language synthesis (NLP), generative modelling, etc.

A "linearized" variant of this idea consists in dropping the activation functions $\sigma_i$, to get a simpler approximation $f(x) \approx W_{k+1}\ldots W_0x$. A largely open problem is understanding the geometry of the values of the weights $W_i$ which give the best possible approximation for a given function $f$.

A recent paper has proposed to use tools from analysis and geometry (mainly *implicit function theorem*, *generalized Sard's theorem*, etc.) to shed some light on the problem. For example, the authors were able to show that:

- For "almost all" $\Lambda \in \mathcal X_+$, every dense $W \in \Omega_\Lambda$ is an isolated point
**[Theorem 1]** - As $\Lambda$ is shrinked uniformly to $0 \in \mathcal X_+$, every point in $\Omega_\Lambda$ either converges to $0 \in \mathcal X$ or diverges to infinity
**[Proposition 1]**.

I'm interesting in knowing whether (and how) algebraic geometry or other educated tools can help produce results in the flavor of the examples above.