Update
The Coursera course I recommended long ago now gone offline. In any case, the field has continued to advance dramatically and there are new results and new expository texts; see any of the more recent answers.
For what it's worth, in the six years since I wrote this answer, the most fruitful point of view in my own work has been to focus on the high-dimensional geometry of neural networks. There are a lot of interesting sites to see in the wilds of a world with thousands or millions of dimensions.
Old answer
If you have time, I highly recommend this Coursera course.
The videos are available for free and are truly excellent. The teacher is Geoffrey Hinton, who is one of the main players in the area, and he does an excellent job of providing both clear definitions and useful intuition.
In general, I wouldn't expect to see perfect theorem-lemma-proof exposition of deep learning anywhere, simply because the math hasn't caught up to real-world practice. More typical is a clean analysis of an idealized system, which is then related to a real system by a heuristic argument. In other words, this is an area that could use attention from mathematicians!