As I see it, the situation is a combination of all of the reasons listed, but I would frame the issues differently:

  1. There are many derivations and topics in physics that are entirely rigorous in principle, but where physicists don't consider it worthwhile to dwell in detail on the issues of rigor.  For instance, what is a "Dirac delta function"?  It's not really a function, but rather a dual vector in the space of continuous functions, but this distinction is often not important in a physics discussion.  This situation is simply a matter of division of labor between mathematicians and physicists.

  2. Many derivations in physics involve ad hoc approximations whose strength is not well understood.  The final equations that are solve are often entirely rigorous, but their relevance to physical reality is negotiable.  For instance, in astronomy, when is it fair to approximate planets as points or spheres?  Or to pick a more serious example, [the shallow water equations][1] are an ad hoc simplification of the Navier-Stokes equations.  This too is a division of labor, but a more troublesome one than in case 1.  A better justification of these ad hoc approximations could certainly be valuable in physics, and some of them are also interesting mathematics conjectures.

  3. Quantum field theory is a special case.  There is very strong evidence that quantum field theory, as understood by methods such as Feynman diagrams but also new methods, is an island of mathematical consistency checks that would ideally be connected to ordinary rigorous mathematics.  Quantum field theory methods can be used for many "theories" that look important for pure mathematics.  Many of these theories have only an abstract resemblance to the true quantum field theories of physics and can't be checked with experiments, but instead satisfy a vast array of consistency checks and lead to many interesting conjectures.  On the other hand, a few of these theories are realistic and do agree with experiments.  There is no good division of labor to explain why quantum field theory isn't rigorous.  It is unfinished business for mathematicians and physicists.

  On the other hand, the mathematical status of quantum field theory has at least improved over time.  Quantum field theory in 1 dimension is rigorous, and significant pieces of conformal field theory in two dimensions (or one complex dimension) are rigorous.

  4. There are also some non-examples that are entirely rigorous but hard to believe.  Relativity is hard to believe and quantum mechanics (in the sense of quantum probability) is hard to believe, but they are entirely rigorous.  They aren't even known to be ad hoc approximations to something else.  (Well, non-quantum general relativity is surely an approximation to some theory of quantum gravity, but let's set that aside for now.)  Sometimes they are explained in a non-rigorous way as in case 1, but that shouldn't fool mathematicians.  It is important not to conflate these rigorous victories with non-rigorous extensions such as quantum field theory and string theory.  Indeed, quantumness is not strictly required to have the disease of case 3; some of the difficulties of rigor already occur for classical stochastic field theories.

  [1]: http://en.wikipedia.org/wiki/Shallow_water_equations