While I do disagree with the premises, some explicit, many implicit, of the question, the question and these premises are surely fairly popular.
While not arguing about whether differential equations and vector fields and tensors "are" physics or not, I would agree that they have huge historical/experiential base of "physical intuition", whether this is "physics" or not, notably.
In fact, it has been found profitable to transport from, or abstract from, physically meaningful situations in "mechanics", say, to "number theory" (as manifest in "automorphic forms", especially). That is, physically unsurprising, if non-trivial, ideas sometimes seem to have non-trivial potential impact on "number theory" suitably translated into harmonic analysis, understandably on special objects, not generic.
A widely-understood cliche, and wonderful it is, is the proof that $\sum 1/n^2=\pi^2/6$ via Plancherel applied to the sawtooth function made periodic, that is, on the circle as $\mathbb R/\mathbb Z$. This is easy to explain, and does touch my aesthetic sense, though I understand it might not touch others'.
If one doubted that the "guts" of differential geometry mattered, I'd volunteer that the proofs that automorphic forms do conform to expectations (about global Sobolev indices and such) do depend on the particulars, so are indeed dependent upon "geometry", whatever our conceits make of the latter.