Here is a list of books I find useful that present some physical topics from a mathematical viewpoint. Sadly I don't know a good reference for electromagnetism, quantum field theory or statistical physics.
Arnold, Mathematical methods of classical mechanics.
Woodhouse, Special relativity.
Woodhouse, General relativity.
Woodhouse, Geometric Quantization.
