The Feynman lectures are good, but one of the main things which separates physics from mathematics is the role of experiment and observation. Physics is not just a matter of getting the formulae and models right, but also of testing mathematical models against observations to see whether they stand up or break down in "the real world". Part of the role of mathematical models is to give physicists some guidance on potentially fruitful places to look.
So it rather depends whether you are looking at mathematical/theoretical physics as a mathematician/theoretician would understand it, or whether you are looking to understand the role of mathematics in physics as a discipline.
