Your question is
Are there ever cases where formally defining physical phenomena in mathematical language is unnecessary?
It is never possible to define physical phenomena directly in mathematical language. Mathematics (and even sciences) can deal with real phenomena only through models.
To deal with a model mathematically, it must be a mathematical model, described in mathematical terms.
I think all mathematical models go back to real needs -- even if very indirectly, through many layers of abstraction and generalization.
So, at least in applications of mathematics, an important goal is to faithfully and efficiently model real phenomena of interest, so as to be able to make accurate enough predictions about the real phenomena.
The clarity and effectiveness of mathematical presentation are of course very important. Therefore, formalization can hardly ever hurt, unless it is taken to such an extent as to actually hamper readers' understanding. The power of formalization is to code very rich, deep, and complex mathematical content into very compact, unambiguously defined, and easy to grasp at once (after some practice) mathematical symbols and formulas.
It is usually helpful if a formal presentation is complemented by appropriate illustrations and imagery. In particular, if a new term is introduced, it helps to give it a descriptive name, easy to remember.
However, if the presentation is very informal and mainly consists of many images each with multiple possible interpretations, then it is easy for the uninitiated reader to quickly get lost (as happened with me many times).
To me, the presentation of deep learning theory linked in your post does not look mathematical -- see e.g. formulas (2.30) and (2.32) on p. 51 there.