Regardless of whether or not it is possible to make specific arguments from physics rigorous, they are often not taught rigorously. I'll give some specifics from my own education.
An obvious example is to consider the content of a basic quantum mechanics course. A course might start by considering wavefunctions. These are asserted to be functions $\mathbb{R}^4\rightarrow\mathbb{C}$ without bothering to state precisely which functions are allowed. Do they need to be continuous, differentiable, smooth, behave in a certain way as we go to infinity? In an elementary course, nobody will bother to say.
We'll be taught that wavefunctions are solutions to PDEs. So we implicitly have to start assuming differentiability to some degree or other. Except we'll be asked to consider potential wells that result in wavefunctions that occasionally have discontinuous derivatives. And then we'll be asked to consider wavefunctions that are Dirac delta functions which are clearly not functions in the usual sense. If you object the lecturer will mutter something about Hilbert spaces under their breath, despite the fact that it's trivial to prove we're being asked to consider spaces of wavefunctions that provably don't form a Hilbert space.
These kinds of elementary quantum mechanics can be made rigorous. But it's not part of the education of many physicists. And this means that actual arguments made by physicists are often not rigorous even when there is no fundamental problem with making them rigorous.
(And much of this sounds like a complaint. But the truth is, back when I was doing physics, I wouldn't have wanted to "waste" time on making these arguments rigorous.)