I certainly can't think of examples similar to your physics examples of concepts that were just wrong so effectively became extinct.

Two extremes which are present in mathematics are

1) Things which become too simple to have their name retain prominence as commonly known terminology. 

For example: For Aristotle, Square numbers and [Oblong number][2]s were perhaps similarly important. Today the first concept is vibrant while second concept is fairly unfamiliar. Similarly the concept of [singly and doubly even][3] integers is fairly unfamiliar being subsumed as the case $p=2$ of the $p$-adic order of an integer (or rational number.)

AND

2) Things which are too complex for the tools of their time and so hibernate for a while

As an example, I feel compelled to quote the stirring first paragraph of [The Invariant Theory of Binary Forms][1] 

*Like the Arabian phoenix rising out of its ashes, the theory of invariants, pronounced dead at the turn of the century, is once again at the forefront of mathematics. During its long eclipse, the language of modern algebra was developed, a sharp tool now at long last being applied to the very purpose for which it was intended. More recently, the artillery of combinatorics began to be aimed at the problems of invariant theory bequeathed to us by the nineteenth century, abandoned at the time because of insufficient combinatorial thrust.*






  [1]: http://www.ams.org/journals/bull/1984-10-01/S0273-0979-1984-15188-7/S0273-0979-1984-15188-7.pdf
  [2]: https://en.wikipedia.org/wiki/Pronic_number
  [3]: https://en.wikipedia.org/wiki/Singly_and_doubly_even