**Note:** *The following is an answer to [this post][1] but everything I posted there applies equally and fully here.* *Let me also comment shortly, that I also found the theory of DNNs difficult to enter since a clear "mathematically formal" point of access is now always clear in broader machine learning literature. Nevertheless, I found these helpful.* ---------- ---------- ---------- **Shallow Feedforward Networks and Deep Convolutional Networks** I would suggest some [Harmonic/Fourier analysis][1], some [constructive approximation theory][2], and their intersection [(esp.: Besov Spaces)][3]. This is because, many of the quantitative approximation theorems for shallow (1-hidden layer) feedforward networks are derived via such methods. Relevant (contemporary) papers for such methods include: - [Approximation spaces of deep neural networks - Gribonval et al. 2021][4] - [Approximation rates for neural networks with general activation functions - Siegel and Xu, 2021][5] - [Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality - Taiji Suzuki][6] This particular point carries over equally to (deep) convolutional networks: - [Universality of deep convolutional neural networks - Ding-Xuan Zhou 2020][7] ---------- **Deep Feedforward Networks and Optimal Rates** Otherwise, for deep feed-forward networks some of the more insightful approximation-theoretic results rely on [Vapnik-Chervonekis Theory][8]. These are then typically used to derive "optimal approximation rates"; see especially these papers: - [Optimal approximation of continuous functions by very deep ReLU networks D. Yarotsky 2018][9] - [The phase diagram of approximation rates for deep neural networks - Dmitry Yarotsky, Anton Zhevnerchuk - 2020][10] ---------- **Non-Euclidean Input/Output Spaces and Topological Embeddings** These results typically rely on results of a more topological flavor. I would [Van Mill's book][11] and of course basic general topology textbooks [like Munkres' classic][12]. The only universal approximation theorems I know of in this context are: - [Non-Euclidean Universal Approximation, 2020][13] - [NEU: A Meta-Algorithm for Universal UAP-Invariant Feature Representation, 2021][14] ---------- **Recurrent Structures and Reservoir Computers** If you're looking for something a bit more "dynamic" in nature, then I would recommend brushing up on your functional analysis, measure theory, and sequences in Banach spaces. The first of these papers makes extensive use of ideas surrounding [Rademacher Complexity][15] and there are deep connections to the [theory of dynamical systems][16]. - [Risk Bounds for Reservoir Computing - Gonon, Grigoryeva, Ortega - 2020][17] - [Differentiable reservoir computing - Grigoryeva, Ortega - 2019][18] I mention here also the developing connections between learning dynamics and [rough path theory][19]. See: - [Deep signature transforms - Kidger, Bonnier, Perez Arribas, Salvi, Lyons - 2019][20] - [Discrete-Time Signatures and Randomness in Reservoir Computing - Cuchiero, Gonon, Grigoryeva, Ortega, Teichmann - 2021][21] ---------- **Qualitative Approximation by Shallow Feedforward Networks "Classical Style"** Let me mention that, classical *(qualitative)* universal approximation results are based on the [Stone-Weierstraß theorem][22] from [approximation theory][23]. Some results rely on the theory of [LF-Spaces][24] which are a class of [Locally-Convex spaces][25] with a particularly ["category-theoretic$\cap$functional-analytic flavor"][26]. For modern formulations of the result in rather general contexts, see: - [Stone–Weierstraß and extension theorems in the nonlocally convex case - Timofte, Timofte, Khan - 2018][27] - [Stone-weierstraß theorems for group-valued functions - Galindo, Sanchis, 2004][28] The last of these references needs only a bit of background in [topological groups][29]. ---------- **Memory Capacity/ Interpolation Capabilities** These results have a variety of backgrounds. The latter of these results draws from the [Chow-Rashevskii Theorem][30] and [control theory][31]. - [Small ReLu networks are powerful memorizers: A tight analysis of memorization capacity - Yun, Sra, Jadbabaie - 2019][32] - [Memory Capacity of Neural Networks with Threshold and Rectified Linear Unit Activations - Vershynin - 2020][33] - [Deep Neural Networks, Generic Universal Interpolation, and Controlled ODEs - Cuchiero, Larsson, Teichmann - 2020][34] ---------- **Impossibility Theorems** Let me briefly round off this post with the following interesting results. The pre-requisits for these papers are a typical background; nonetheless, their results are fascinating. - [Minimum Width for Universal Approximation - Park, Yun, Lee, Shin - 2021][35] - [Deep, Skinny neural networks are not universal approximators - Johnson - 2018][36] [1]: https://en.wikipedia.org/wiki/Harmonic_analysis [2]: https://en.wikipedia.org/wiki/Constructive_Approximation [3]: https://en.wikipedia.org/wiki/Besov_space [4]: https://link.springer.com/article/10.1007/s00365-021-09543-4 [5]: https://www.sciencedirect.com/science/article/pii/S0893608020301891#! [6]: https://openreview.net/forum?id=H1ebTsActm [7]: https://www.sciencedirect.com/science/article/pii/S1063520318302045 [8]: https://en.wikipedia.org/wiki/Vapnik%E2%80%93Chervonenkis_theory [9]: http://proceedings.mlr.press/v75/yarotsky18a [10]: https://papers.nips.cc/paper/2020/hash/979a3f14bae523dc5101c52120c535e9-Abstract.html [11]: http://%20The%20Infinite-Dimensional%20Topology%20of%20Function%20Spaces [12]: https://www.pearson.com/us/higher-education/product/Munkres-Topology-2nd-Edition/9780131816299.html [13]: https://proceedings.neurips.cc/paper/2020/file/786ab8c4d7ee758f80d57e65582e609d-Paper.pdf [14]: https://www.jmlr.org/papers/v22/18-803.html [15]: https://en.wikipedia.org/wiki/Rademacher_complexity [16]: https://en.wikipedia.org/wiki/Dynamical_systems_theory [17]: https://www.jmlr.org/papers/volume21/19-902/19-902.pdf [18]: https://www.jmlr.org/papers/volume20/19-150/19-150.pdf [19]: https://en.wikipedia.org/wiki/Rough_path [20]: https://papers.nips.cc/paper/2019/hash/d2cdf047a6674cef251d56544a3cf029-Abstract.html [21]: https://ieeexplore.ieee.org/document/9442205 [22]: https://en.wikipedia.org/wiki/Stone%E2%80%93Weierstrass_theorem [23]: https://en.wikipedia.org/wiki/Approximation_theory [24]: https://en.wikipedia.org/wiki/LF-space [25]: https://en.wikipedia.org/wiki/Locally_convex_topological_vector_space [26]: https://mathoverflow.net/questions/83363/is-there-a-nice-application-of-category-theory-to-functional-complex-harmonic-an [27]: https://www.sciencedirect.com/science/article/pii/S0022247X18301823 [28]: https://link.springer.com/article/10.1007/BF02772227 [29]: https://ncatlab.org/nlab/show/topological+group [30]: https://en.wikipedia.org/wiki/Chow%E2%80%93Rashevskii_theorem [31]: https://en.wikipedia.org/wiki/Control_theory [32]: https://papers.nips.cc/paper/2019/hash/dbea3d0e2a17c170c412c74273778159-Abstract.html [33]: https://epubs.siam.org/doi/pdf/10.1137/20M1314884 [34]: https://epubs.siam.org/doi/abs/10.1137/19M1284117 [35]: https://openreview.net/forum?id=O-XJwyoIF-k [36]: https://openreview.net/forum?id=ryGgSsAcFQ