I have recently become interested in Machine Learning and AI as a student of theoretical physics and mathematics, and have gone through some of the recommended resources dealing with statistical learning theory and deep learning in neural networks.

One of the main issues I find in my personal study of this discipline is that an overwhelming proportion of such resources focus on the practical side of ML, forfeiting rigour in favour of useful heuristics. This approach has its obvious merits, considering the great current interest in their applications both in science and technology, but I would like to go beyond what the average engineer might need and explore the more theoretical sides.

The elephant in the room is, of course, the fact that to date the inner workings of the main tools of AI, neural networks above all others, are not well understood. From what I can tell, there are a variety of approaches drawing from very diverse field, including a physical perspective (see Huang's Statistical Mechanics of Neural Networks, or Statistical Field Theory for Neural Networks by Helias and Dahmen).

As an outsider, I have a hard time navigating the literature, so I have thought to ask quite an open question on this site (whether this is the right place I do not know; I'm sure moderation will let me know if it isn't). Could anyone lay out a map of the current landscape of AI research, from mainstream science to the cutting-edge approaches, and elucidate the types of mathematics needed to tackle them?