I just found this paper-https://arxiv.org/pdf/1801.05894.pdf, which introduces deep learning in a mathematically sound manner, specially for computations of backpropagation etc. As a mathematician who worked in machine learning but not deep learning, I've noticed that the tools that're often needed for machine learning are linear algebra (with a bit of functional analysis), probability and optimization.

Apart from the above, there're also some literature using differential geometry and triangle meshes, e.g. https://arxiv.org/pdf/1611.08097.pdf, although I'm not quite sure whether they're being used in industrial applications. My suspicion comes from the fact that in general most of machine learning that uses differential geometry, e.g. manifold learning, is mostly useless outside academia, hence in real world, as real life observations don't satisfy smooth manifold assumption. However, I'd be glad, as a mathematician, to be proven wrong, in the sense if you could point out some true industrial use of geometric machine learning/geometric deep learning to solve realistic problems.