First of all "physics" is rather general.  You're more likely to find good books on more specific topics like special relativity, quantum mechanics, etc.

Second, if you have the time I would encourage you to read physics books that are written _for physicists_, not for mathematicians.  There are numerous differences in terminology and worldview between the physics and mathematics community, even when the underlying subject matter is in some sense the same.  It's very valuable for a mathematician to be able to read and understand recent physics arxiv postings, and the only way to do this is to go through some (perhaps accelerated) version of physics grad school.

Here are some physics books which I have enjoyed.  The list is of course constrained by my own limited experience.

* Electricity and Magnetism, Berkeley Physics Course Vol. II by Edward M. Purcell.  This book presupposes knowledge of special relativity, but I thought is was really great when I read it as an undergraduate.

* Feynman lectures on physics.  Not mathematically sophisticated, but very readable and also covers many different topics.

* The Quantum Theory of Fields, volume 1 by Steven Weinberg.  I found this book to be much less impenetrable (from the point of view of a mathematician who foolishly stopped taking physics courses when he was an undergraduate) than the typical QFT textbook.

* Quantum Field Theory in a Nutshell by Anthony Zee.  This book omits a lot of details and emphasizes the big picture.  It's a great companion to a more detailed book on QFT.