I would like to know what mathematics topics are the most important to learn before actually studying the theory on neural networks.
I ask that because I will start to learn about neural networks and machine learning on my own to help in the analysis I am doing on my PhD about patterns of genome evolution.
Thank you in advance.
For basic neural networks (i.e. if you just need to build and train one), I think basic calculus is sufficient, maybe things like gradient descent and more advanced optimization algorithms. For more advanced topics in NNs (convergence analysis, links between NNs and SVMs, etc.), somewhat more advanced calculus may be needed.
For machine learning, mostly you need to know probability/statistics, things like Bayes theorem, etc.
Since you are a biologist, I don't know whether you studied linear algebra. Some basic ideas from there are definitely extremely useful. Specifically, linear transformations, diagonalization, SVD (that's related to PCA, which is a pretty basic method for dimensionality reduction).
The book by Duda/Hart/Stork has several appendices which describe the basic math needed to understand the rest of the book.
I think the answer depends on which structure you are looking to approximate, and, in what sense you want to approximate it. Below, you'll find a few contemporary references to help out :)
Shallow Feedforward Networks and Deep Convolutional Networks
I would suggest some Harmonic/Fourier analysis, some constructive approximation theory, and their intersection (esp.: Besov Spaces). This is because, many of the quantitative approximation theorems for shallow (1-hidden layer) feedforward networks are derived via such methods. Relevant (contemporary) papers for such methods include:
Deep Feedforward Networks and Optimal Rates
Otherwise, for deep feed-forward networks some of the more insightful approximation-theoretic results rely on Vapnik-Chervonekis Theory. These are then typically used to derive "optimal approximation rates"; see especially these papers:
Non-Euclidean Input/Output Spaces and Topological Embeddings
These results typically rely on results of a more topological flavor. I would van Mill's book and, of course, basic general topology textbooks like Munkres' classic. The only universal approximation theorems I know of in this context are:
Recurrent Structures and Reservoir Computers
If you're looking for something a bit more "dynamic" in nature, then I would recommend brushing up on your functional analysis, measure theory, and sequences in Banach spaces. The first of these papers makes extensive use of ideas surrounding Rademacher Complexity and there are deep connections to the theory of dynamical systems.
I mention here also the developing connections between learning dynamics and rough path theory. See:
Qualitative Approximation by Shallow Feedforward Networks "Classical Style"
Let me mention that, classical (qualitative) universal approximation results are based on the Stone-Weierstraß theorem from approximation theory. Some results rely on the theory of LF-Spaces which are a class of Locally-Convex spaces with a particularly "category-theoretic$\cap$functional-analytic flavor". For modern formulations of the result in rather general contexts, see:
The last of these references needs only a bit of background in topological groups.
Memory Capacity/ Interpolation Capabilities These results have a variety of backgrounds. The latter of these results draws from the Chow-Rashevskii Theorem and control theory.
Impossibility Theorems Let me briefly round off this post with the following interesting results. The pre-requisits for these papers are a typical background; nonetheless, their results are fascinating.
I believe Ian Goodfellow and Yoshua Bengio's Deep Learning book covers the basics and also how you would use it for research. The chapters are also available online for free.
Take a look at the web page for Michael Steele's course Probability Inequalities and Machine Learning, and the various texts linked to there.
As a deep learning practitioner with mathematical background I was yearning to have some satisfying mathematical framework of what I do in my every day job. In my opinion, very well fitted mathematical foundations of deep learning principles are captured simply by Empirical risk minimization (ERM) concept. I encourage everyone to read just Chapter 1 from The Nature of Statistical Learning Theory of V. Vapnik. In my opinion it is an eye opener.
I recently wrote an answer to a related question, about math research that can enhance machine learning. But, part of what I wrote is also related to a learning resource that might help someone interested in neural networks. That resource is the book Data Science for Mathematicians, edited by Nathan Carter. It assumes the audience is a mathematician (at, say, the graduate student level), then gives high level treatments of:
I should disclose that I wrote one of the chapters, but don't have any financial stake in the book. I recommend it because I think it's great, and will help mathematicians who want to embrace data science in their research, teaching, or as an alternative career. I hope it helps!
