If you really know nothing about physics I suggest you begin with any text book on physics for undergrad. Easy to read, it will introduce the main usual suspects. After, you'll ask again :)
I am not sure that jumping from nothing to quantum mechanics, or even worse quantum fields theory, would be wise, like jumping from nothing in math to algebraic geometry or K-Theory.
After that, it depends of course at what level of mathematical physics you want to stop. I will illustrate this with some examples:
Question: What is the "mass" of an isolated dynamical system?
Math Answer: It is the class of cohomology of the action of the group of Galilee, measuring the lack of equivariance of the moment map, on a symplectic manifold representing the isolated dynamical system.
Another question: Why in general relativity $E = mc^2$?
Math Answer: Because the group of Poincaré has no cohomology
Another, other question: What is the theorem of decomposition of motions around the center of gravity?
Math Answer: Let $(M,\omega)$ be a symplectic manifold with an hamiltonian action of the group of Galilee, if the "mass" of the system is not zero (in the sense above) then $M$ is the symplectic product or $({\bf R}^6, {\rm can})$, representing the motions of the center of gravity, by another symplectic manifold $(M_0,\omega_0)$, representing the motions around the center of gravity. The group of Galillee acting naturally on $\bf R^6$ and $SO(3) \times {\bf R}$ on $M_0$.
Another, other, other question: What are the constants of motions?
Math Answer: Let $(M,\omega)$ be a pre-symplectic manifold with an hamiltonian action of a Lie group $G$, then the moment map is constant on the characteristics of $\omega$, that is the integral manifolds of the vector distribution $x \mapsto \ker(\omega_x)$.
These answers are the mathematical versions of physics classical constructions, but it would be very difficult to appreciate them if you have no pedestrian introduction of physics. You may enjoy also Aristotles' book "Physics", as a first dish, just for tasting the flavor of physics :)
After that, you will be able to appreciate also quantum mechanics, but this is another question.
Addendum
Just before entering in the modern world of physics I would suggest few basic lectures for the winter evenings, near the fireplace (I'm sorry I write them down in french because I read them in french).
• Platon, Timée, trad. Émile Chambry.
• Aristote, La Physique, Éd. J. Vrin.
• Maïmonide, Le Guide des Égarés, Éd. Maisonneuve & Larose. (the part about time as an accident of motion, accident of the thing. Very deep and modern thoughts).
• Giordano Bruno, Le Banquet des Cendres, Éd. L’éclat.
• Galileo Galilei, Dialogue sur les Deux Grands Systèmes du Monde, Éd. Points.
• Albert Einstein, La Relativité, Éd. Payot.
• Joseph-Louis Lagrange, Mécanique Analytique, Éd. Blanchard.
• Felix Klein, Le Programme d’Erlangen, Éd. Gauthier-Villars.
• Jean-Marie Souriau, Structure des Systèmes Dynamiques, Éd. Dunod.
• Victor Guillement & Shlomo Sternberg, Geometric Asymptotics, AMS Math Books
• François DeGandt Force and Geometry in Newton Principia.