28
$\begingroup$

I would like to know what mathematics topics are the most important to learn before actually studying the theory on neural networks.

I ask that because I will start to learn about neural networks and machine learning on my own to help in the analysis I am doing on my PhD about patterns of genome evolution.

Thank you in advance.

$\endgroup$
2
  • 6
    $\begingroup$ Marcos, machine learning is still very young, and in a state of great flux. What this means is that the techniques used are constantly developing, and if you looked across the whole spectrum, you'd see a dizzying variety of methods. What I recommend is to google around for courses on topics you are interested in, both at the school you will attend and at others you respect, and see what the textbooks, notes, and homework assignments look like. $\endgroup$ Jan 15, 2010 at 10:15
  • $\begingroup$ Mathematical Logic is the most basic subject you need to learn. $\endgroup$
    – markvs
    Aug 10, 2021 at 22:17

6 Answers 6

13
$\begingroup$

For basic neural networks (i.e. if you just need to build and train one), I think basic calculus is sufficient, maybe things like gradient descent and more advanced optimization algorithms. For more advanced topics in NNs (convergence analysis, links between NNs and SVMs, etc.), somewhat more advanced calculus may be needed.

For machine learning, mostly you need to know probability/statistics, things like Bayes theorem, etc.

Since you are a biologist, I don't know whether you studied linear algebra. Some basic ideas from there are definitely extremely useful. Specifically, linear transformations, diagonalization, SVD (that's related to PCA, which is a pretty basic method for dimensionality reduction).

The book by Duda/Hart/Stork has several appendices which describe the basic math needed to understand the rest of the book.

$\endgroup$
12
$\begingroup$

I think the answer depends on which structure you are looking to approximate, and, in what sense you want to approximate it. Below, you'll find a few contemporary references to help out :)


Shallow Feedforward Networks and Deep Convolutional Networks

I would suggest some Harmonic/Fourier analysis, some constructive approximation theory, and their intersection (esp.: Besov Spaces). This is because, many of the quantitative approximation theorems for shallow (1-hidden layer) feedforward networks are derived via such methods. Relevant (contemporary) papers for such methods include:


Deep Feedforward Networks and Optimal Rates

Otherwise, for deep feed-forward networks some of the more insightful approximation-theoretic results rely on Vapnik-Chervonekis Theory. These are then typically used to derive "optimal approximation rates"; see especially these papers:


Non-Euclidean Input/Output Spaces and Topological Embeddings

These results typically rely on results of a more topological flavor. I would Von Mill's book and of course basic general topology textbooks like Munkres' classic. The only universal approximation theorems I know of in this context are:


Recurrent Structures and Reservoir Computers

If you're looking for something a bit more "dynamic" in nature, then I would recommend brushing up on your functional analysis, measure theory, and sequences in Banach spaces. The first of these papers makes extensive use of ideas surrounding Rademacher Complexity and there are deep connections to the theory of dynamical systems.

I mention here also the developing connections between learning dynamics and rough path theory. See:


Qualitative Approximation by Shallow Feedforward Networks "Classical Style"

Let me mention that, classical (qualitative) universal approximation results are based on the Stone-Weierstraß theorem from approximation theory. Some results rely on the theory of LF-Spaces which are a class of Locally-Convex spaces with a particularly "category-theoretic$\cap$functional-analytic flavor". For modern formulations of the result in rather general contexts, see:

The last of these references needs only a bit of background in topological groups.


Memory Capacity/ Interpolation Capabilities These results have a variety of backgrounds. The latter of these results draws from the Chow-Rashevskii Theorem and control theory.


Impossibility Theorems Let me briefly round off this post with the following interesting results. The pre-requisits for these papers are a typical background; nonetheless, their results are fascinating.

$\endgroup$
2
  • 1
    $\begingroup$ These references look good for many people on this site, but probably not for someone like the OP doing a PhD about patterns of genome evolution. $\endgroup$
    – user44143
    Jun 29, 2021 at 12:26
  • $\begingroup$ Fair enough, but the thread can still be useful to some other readers (or if the OP needs some more technical details or estimates). $\endgroup$
    – ABIM
    Jun 29, 2021 at 12:50
6
$\begingroup$

I believe Ian Goodfellow and Yoshua Bengio's Deep Learning book covers the basics and also how you would use it for research. The chapters are also available online for free.

$\endgroup$
5
$\begingroup$

Take a look at the web page for Michael Steele's course Probability Inequalities and Machine Learning, and the various texts linked to there.

$\endgroup$
2
  • 3
    $\begingroup$ That page seems to be more related to what's called "learning theory", i.e. theoretical (mathematical) analysis of various learning algorithms, their performance bounds (hence probability inequalities), etc. $\endgroup$
    – user3035
    Jan 15, 2010 at 0:36
  • $\begingroup$ Just a comment. Imo learning theory is more statistics than mathematics. For example, the approximation theory of various learning models isn't learning theory while it's applied analysis. $\endgroup$
    – ABIM
    Jan 9, 2023 at 0:41
4
$\begingroup$

As a deep learning practitioner with mathematical background I was yearning to have some satisfying mathematical framework of what I do in my every day job. In my opinion, very well fitted mathematical foundations of deep learning principles are captured simply by Empirical risk minimization (ERM) concept. I encourage everyone to read just Chapter 1 from The Nature of Statistical Learning Theory of V. Vapnik. In my opinion it is an eye opener.

$\endgroup$
0
$\begingroup$

I recently wrote an answer to a related question, about math research that can enhance machine learning. But, part of what I wrote is also related to a learning resource that might help someone interested in neural networks. That resource is the book Data Science for Mathematicians, edited by Nathan Carter. It assumes the audience is a mathematician (at, say, the graduate student level), then gives high level treatments of:

  • programming with data,
  • linear algebra (and its applications to data analytics),
  • basic statistics,
  • clustering,
  • operations research,
  • dimensionality reduction,
  • machine learning,
  • deep learning, and
  • topological data analysis

I should disclose that I wrote one of the chapters, but don't have any financial stake in the book. I recommend it because I think it's great, and will help mathematicians who want to embrace data science in their research, teaching, or as an alternative career. I hope it helps!

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.