Placing the authors out-of-order in a mathematics paper makes a strong statement -- that one author has contributed significantly more than another. There are problems with the alphabetical system, and there are also problems with the ordered-by-contribution system, e.g. when authors contribute comparable amounts to a paper, who comes first?

To be fair, the proportion of papers that have authors out-of-order should be contrasted with the likelihood of a random permutation of those authors' names being out-of-order. So, we should disregard papers with a single author. If there's two authors, then there's a 0.5 probability that "alphabetical order" = "ordered by contribution". Then we need to keep in mind that there's fewer papers with 3 or more authors.

There are examples (not just famous ones) around in the mathematics journals if you look for them (I'm guessing often people wouldn't even notice that they're out of alphabetical order). My former supervisor has two:

S. Taylor, I. M. Wanless and N. L. Boland, Distance domination and amplifier placement problems, Australas. J. Combin. 34 (2006) 117-136.

I. M. Wanless and E. C. Ihrig, Symmetries that Latin squares inherit from 1-factorizations, J. Combin. Des., 13 (2005) 157-172.