One thing that comes directly to mind is the calculus of variations, in the classical sense, where the point is to get rigorous results by mathematical analysis.

Now, there are probably several typical kinds of objection here. Firstly the area really is not dormant: physicists use it in the same fashion as ever; there are various kinds of "variational formalism" discussed, for example in soliton theory; and mathematicians have "gone round" this area by the use of Morse theory and moment maps, to break out of the traditional formulation into areas of geometry.

But in terms of identifying a "break in tradition" (I don't entirely agree with Quinn's framing of the issue, but it is real enough when those who wrote the papers are no longer around) I would guess there is no line of textbooks that continues from the early twentieth century treatments. Few people may know what was considered important in that line of development. I'm aware of work on variational problems (e.g. the Plateau problem) that is pretty much current, but that illustrates one tendency, to make a given problem into a theory of its own. Anyway, do mathematicians in general know why Jesse Douglas got a Fields Medal in 1936? How many could read his papers?