Mathematics is virtually the only profession that has the luxury of insisting on near-100% certainty. In physics, one is always making approximations and idealised assumptions, and even the known laws of physics may themselves only be approximations of a more accurate set of laws. In such a context, it is far more efficient to rely on heuristics that are accurate 99% of the time, than to insist on rigorous proof that is correct 100% of the time. (In principle, even a mathematically rigorous argument could be subject to scrutiny by the logicians and philosophers, but this is still a far stricter standard than what is needed for a physically convincing argument.)
A typical example of a heuristic is this: if a dynamical system has no obvious conservation laws or other structure to it that would restrict the dynamics, then it is likely to be mixing. With this heuristic one can do a huge amount of statistical mechanics; but it is completely non-rigorous. And indeed, the rigorous theory of statistical mechanics lags far behind the heuristic theory, and is thus far less useful for physics. In physics, it's OK if Maxwell's demon (or some other troublemaker) steps in to ruin everything $10^{-100}$ of the time; but this is unacceptable by the standards of rigorous mathematics.
A mathematical instance of the above heuristic in action would be the assertion that the digits of pi are uniformly distributed because there is no obvious reason why they should not be so. This is extremely convincing at the heuristic level, but falls well short of a rigorous proof, which is still completely out of reach of known methods (we can't rule out the bizarre possibility that pi has attached to it its own "Maxwell's demon").
Note though that these types of heuristics are routinely used in certain areas of mathematics, such as cryptography; and conversely there are plenty of physicists who work on rigorous mathematics; and there are people who would classify themselves as both mathematicians and physicists, or who do a mixture of rigorous and heuristic work. So the distinction is not really so sharp on closer inspection.
