# Understanding a certain algebraic set arising in Deep Learning

I'm not a professional geometer. Thanks in advance for your patience.

So, let $$n$$, $$k$$, $$p_0,\ldots,p_{k}$$ be positive integers. Let $$X$$ (resp. $$Y$$) be an $$p_0$$-by-$$n$$ (resp. an $$p_{k}$$-by-$$n$$) real matrix. Consider the algebraic set $$\Omega := \{(W,\Lambda) \in \mathcal R \times \mathcal X_+ | F_i(W,\Lambda) = 0,\;\forall i=0,\ldots,k\},$$ where

• $$F_i(W,\Lambda):=U_{i+1}^T(V_0XX^T-YX^T)V_{i+1}^T + \Lambda_i \circ W_i$$, a polynomial in the matrices $$W_j$$, and in the entries of the matrices $$W_j$$ and $$\Lambda_j$$.
• $$U_i := \Pi_{j=k}^{i}W_i\in \mathbb R^{p_i \times p_k}$$,
• $$V_i := \Pi_{j=i}^{k}W_i \in \mathbb R^{p_k \times p_i}$$,
• $$\mathcal R := \mathbb R^{p_0 \times p_1} \times \ldots \times \mathbb R^{p_{k-1} \times p_{k}}$$, and
• $$\mathcal X_+ := \mathbb R_+^{p_0 \times p_1} \times \ldots \times \mathbb R_+^{p_{k-1} \times p_{k}}$$

Finally, for any $$\Lambda \in \mathcal X_+$$, define $$\Omega_\Lambda := \{W|(W,\Lambda) \in \Omega\}$$.

# Note

In defining the auxiliary set $$\mathcal R$$, I've taken the base field to be the real numbers $$\mathbb R$$. It might be that a cleaner picture is obtained by looking at a larger base object (e.g the complex numbers $$\mathbb C$$, etc.). Please feel free to change this to suite your needs.

Defintion. A dense matrix is a matrix with only nonzero entries.

# Question

• What can be said about the geometric / topological structure of $$\Omega_\Lambda$$ (resp. of $$\Omega$$) ?
• How many isolated points are there ? How many dense isolated points ? (in either case, a good upper bound is ok :) )
• What are the connected pieces of this set ?
• etc.
• What can be said about the asymptotic geoemetric / topological structure of $$\Omega_\Lambda$$ in the limit $$\Lambda \rightarrow 0$$ ?

# Motivation

There is a rapidly growing branch of statistics called Deep Learning (ok most statisticians might disown it, but let's take this for granted). The idea is to approximate "any" function $$f(x)$$ via a product of matrices $$W_i$$, punctuated with nonlinear so-called "activation functions" $$\sigma_i$$ (a typical choice are Rectified Linear Units or ReLUs for short, defined by $$\sigma_i(x):=\max(x,0)$$). succinctly, $$f(x) \approx \sigma_{k+1}(W_{k+1}(\sigma_k(\ldots(\sigma_0(W_0 x))\ldots)$$. The matrices $$W_i$$ are the parameters of the model, and a optimized for a given task (this is curve-fitting).

Implementations of this simple idea yield super-human performance on certain task, including: computer vision (object recognition, etc.) and language synthesis (NLP), generative modelling, etc.

A "linearized" variant of this idea consists in dropping the activation functions $$\sigma_i$$, to get a simpler approximation $$f(x) \approx W_{k+1}\ldots W_0x$$. A largely open problem is understanding the geometry of the values of the weights $$W_i$$ which give the best possible approximation for a given function $$f$$.

A recent paper has proposed to use tools from analysis and geometry (mainly implicit function theorem, generalized Sard's theorem, etc.) to shed some light on the problem. For example, the authors were able to show that:

• For "almost all" $$\Lambda \in \mathcal X_+$$, every dense $$W \in \Omega_\Lambda$$ is an isolated point [Theorem 1]
• As $$\Lambda$$ is shrinked uniformly to $$0 \in \mathcal X_+$$, every point in $$\Omega_\Lambda$$ either converges to $$0 \in \mathcal X$$ or diverges to infinity [Proposition 1].

I'm interesting in knowing whether (and how) algebraic geometry or other educated tools can help produce results in the flavor of the examples above.

• Just an observation: implicit function theorem and Sard's theorem are more differential topology than algebraic geometry. Also, the implicit function theorem isn't true in AG land (unless you consider its formal power series variant in the case of smooth varieties). – Qfwfq Nov 6 '18 at 20:33
• @Qfwfq Thanks for the comment. Will edit the question to reflect that. – dohmatob Nov 6 '18 at 20:36
• There are too many variables here for me to get a grip on this problem easily, even though I am familiar with the alternation of matrices and nonlinear activators. Can you give the flavor with a simpler special case as an example? – Matt F. Nov 7 '18 at 0:40
• @MattF. There are a few transparent examples in the reference paper arxiv.org/pdf/1810.07716.pdf. Notably examples 1, 2, and 3 (page 5). I'll update the question asap, to direct embed these examples. – dohmatob Nov 7 '18 at 14:19
• @MattF. I've started breaking down the question into simpler pieces. First piece is here mathoverflow.net/questions/314948/…. Thanks. – dohmatob Nov 9 '18 at 17:18