Note: The following is an answer to this post but everything I posted there applies equally and fully here.
Let me also comment shortly, that I also found the theory of DNNs difficult to enter since a clear "mathematically formal" point of access is now always clear in broader machine learning literature. Nevertheless, I found these helpful.
Shallow Feedforward Networks and Deep Convolutional Networks
I would suggest some Harmonic/Fourier analysis, some constructive approximation theory, and their intersection (esp.: Besov Spaces). This is because, many of the quantitative approximation theorems for shallow (1-hidden layer) feedforward networks are derived via such methods. Relevant (contemporary) papers for such methods include:
- Approximation spaces of deep neural networks - Gribonval et al. 2021
- Approximation rates for neural networks with general activation functions - Siegel and Xu, 2021
- Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality - Taiji Suzuki This particular point carries over equally to (deep) convolutional networks:
- Universality of deep convolutional neural networks - Ding-Xuan Zhou 2020
Deep Feedforward Networks and Optimal Rates
Otherwise, for deep feed-forward networks some of the more insightful approximation-theoretic results rely on Vapnik-Chervonekis Theory. These are then typically used to derive "optimal approximation rates"; see especially these papers:
- Optimal approximation of continuous functions by very deep ReLU networks D. Yarotsky 2018
- The phase diagram of approximation rates for deep neural networks - Dmitry Yarotsky, Anton Zhevnerchuk - 2020
Non-Euclidean Input/Output Spaces and Topological Embeddings
These results typically rely on results of a more topological flavor. I would Van Mill's book and of course basic general topology textbooks like Munkres' classic. The only universal approximation theorems I know of in this context are:
- Non-Euclidean Universal Approximation, 2020
- NEU: A Meta-Algorithm for Universal UAP-Invariant Feature Representation, 2021
Recurrent Structures and Reservoir Computers
If you're looking for something a bit more "dynamic" in nature, then I would recommend brushing up on your functional analysis, measure theory, and sequences in Banach spaces. The first of these papers makes extensive use of ideas surrounding Rademacher Complexity and there are deep connections to the theory of dynamical systems.
- Risk Bounds for Reservoir Computing - Gonon, Grigoryeva, Ortega - 2020
- Differentiable reservoir computing - Grigoryeva, Ortega - 2019
I mention here also the developing connections between learning dynamics and rough path theory. See:
- Deep signature transforms - Kidger, Bonnier, Perez Arribas, Salvi, Lyons - 2019
- Discrete-Time Signatures and Randomness in Reservoir Computing - Cuchiero, Gonon, Grigoryeva, Ortega, Teichmann - 2021
Qualitative Approximation by Shallow Feedforward Networks "Classical Style"
Let me mention that, classical (qualitative) universal approximation results are based on the Stone-Weierstraß theorem from approximation theory. Some results rely on the theory of LF-Spaces which are a class of Locally-Convex spaces with a particularly "category-theoretic$\cap$functional-analytic flavor". For modern formulations of the result in rather general contexts, see:
- Stone–Weierstraß and extension theorems in the nonlocally convex case - Timofte, Timofte, Khan - 2018
- Stone-weierstraß theorems for group-valued functions - Galindo, Sanchis, 2004
The last of these references needs only a bit of background in topological groups.
Memory Capacity/ Interpolation Capabilities These results have a variety of backgrounds. The latter of these results draws from the Chow-Rashevskii Theorem and control theory.
- Small ReLu networks are powerful memorizers: A tight analysis of memorization capacity - Yun, Sra, Jadbabaie - 2019
- Memory Capacity of Neural Networks with Threshold and Rectified Linear Unit Activations - Vershynin - 2020
- Deep Neural Networks, Generic Universal Interpolation, and Controlled ODEs - Cuchiero, Larsson, Teichmann - 2020
Impossibility Theorems Let me briefly round off this post with the following interesting results. The pre-requisits for these papers are a typical background; nonetheless, their results are fascinating.
