Replication crisis in mathematics

Question

Lately, I have been learning about the replication crisis, see How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth (good YouTube video) — by Michael Shermer and Stuart Ritchie. According to Wikipedia, the replication crisis (also known as the replicability crisis or reproducibility crisis) is

an ongoing methodological crisis in which it has been found that many scientific studies are difficult or impossible to replicate or reproduce. The replication crisis affects the social sciences and medicine most severely.

Has the replication crisis impacted (pure) mathematics, or is mathematics unaffected? How should results in mathematics be reproduced? How can complicated proofs be replicated, given that so few people are able to understand them to begin with?

Answer 1

Mathematics does have its own version of the replicability problem, but for various reasons it is not as severe as in some scientific literature.

A good example is the classification of finite simple groups - this was a monumental achievement (mostly) completed in the 1980's, spanning tens of thousands of pages written by dozens of authors. But over the past 20 years there has been significant ongoing effort undertaken by Gorenstein, Lyons, Solomon, and others to consolidate the proof in one place. This is partially to simplify and iron out kinks in the proof, but also out of a very real concern that the proof will be lost as experts retire and the field attracts fewer and fewer new researchers. This is one replicability issue in mathematics: some bodies of mathematical knowledge slide into folklore or arcana unless there is a concerted effort by the next generation to organize and preserve them.

Another example is the ongoing saga of Mochizuki's proposed proof of the abc conjecture. The proof involve thousands of pages of work that remains obscure to all but a few, and there remains serious disagreement over whether the argument is correct. There are numerous other examples where important results are called into question because few experts spend the time and energy necessary to carefully work through difficult foundational theory - symplectic geometry provides another recent example.

Why do I think these issues are not as big of a problem for mathematics as analogous issues in the sciences?

1. Negative results: If you set out to solve an important mathematical problem but instead discover a disproof or counterexample, this is often just as highly valued as a proof. This provides a check against the perverse incentives which motivate some empirical researchers to stretch their evidence for the sake of getting a publication.
2. Interconnectedness: Most mathematical research is part of an ecosystem of similar results about similar objects, and in an area with enough activity it is difficult for inconsistencies to develop and persist unnoticed.
3. Generalization: Whenever there is a major mathematical breakthrough it is normally followed by a flurry of activity to extend it and solve other related problems. This entails not just replicating the breakthrough but clarifying it and probing its limits - a good example of this is all the work in the Langlands program which extends and clarifies Wiles' work on the modularity theorem.
4. Purity: social science and psychology research is hard because the results of an experiment depend on norms and empirical circumstances which can change significantly over time - for instance, many studies about media consumption before the 90's were rendered almost irrelevant by the internet. The foundations of an area of mathematics can change, but the logical correctness of a mathematical argument can't (more or less).

Answer 2

How can we expect that increasingly complicated proofs are replicated when so few people can understand them in the first place?

My answer to that is that we do not expect them to be replicated in the usual sense of this word (repeated and included into textbooks with just minor cosmetic and stylistic changes). Rather we expect them to be gradually simplified and streamlined either through changing the proofs themselves by finding a shortcut or replacing the whole argument with a completely different one, or by building a theory that is locally trivial but proceeds in the direction of making the proof understandable and verifiable much faster than the currently existing one. The latter is exactly what Mochizuki tried to do though his goal was rather to just reduce the difficulty from "totally impossible" to "barely feasible" and the prevailing opinion is that he failed in the case of the ABC conjecture though he has succeeded in several other problems.

The first approach is more common in analysis (broadly understood), the second is more common in algebra (also broadly understood), but you can try to play either game in either field. My own perception of what is proved and what is not borders on solipsism: I accept the fact as proven if I've read and understood the whole argument or figured it out myself. So most mathematics remains "unproved" to me and, apparently, will stay unproved for the rest of my life. Of course, it doesn't mean that I'm running around questioning the validity of the corresponding theorems. What it means is that I just never allow myself to rely in my own papers on anything that I haven't fully verified to my satisfaction, try to make my papers as self-contained as possible within practical limits, and that I consider the activity of simplifying the existing proofs as meaningful as solving open questions even in the case when the proofs are reasonably well-known and can already be classified as "accessible". But not everybody works this way. Many people are completely happy to drop a nuke any time they have an opportunity to do it and there is nothing formally wrong with that: the underlying point of view is that our time is short, we have to figure out as many things as possible, and the simplifications, etc. will come later. Probably, we need a mixture of both types to proceed as efficiently as we can.

So I would say that the mathematics is reasonably immune to this crisis in the sense that mathematicians are aware of the associated risks, take them willingly, and try to gradually build the safe ground of general accessibility under everything though the process of this building is always behind the process of the mathematical discovery itself. The same applies to physics and medicine though the gap between the "front line" and the "safe ground" there may be wider. In fact, it applies to any science that deserves to be called by that name. As to the so called "social sciences", they are often done at the level of alchemy and astrology today in my humble opinion (and not only mine: read the Richard Feinman critiques, for example) but we should not forget that those were the precursors to such well-respected sciences as chemistry and astronomy/cosmology, so I view the current crisis there as a part of the normal healthy process of transitioning from the prevailing general "blahblahblah" and weathervane behavior with respect to political winds to something more substantial.

Edit: Paul Siegel has convinced me that things have indeed changed since the time I took (obligatory) courses of Marxist philosophy and the history of communist party, though this change may be not easily visible to the general public because it mainly happens outside academia and is driven primarily by company business interests, so a huge part of it occurs behind closed doors (Paul, please correct me if I misinterpreted what you said in any way). So my statement that the current social sciences are not capable of something beyond general blahblahblah is no longer valid and I retract it. However I still maintain the opinion that it is blahblahblah rather than hard data analysis or other scientific approach that drives many public political and social discussions and decisions of today (I don't know what happens here behind the closed doors, of course, and it may be that, like in advertising, what we see is just what shepherds choose to show to their sheep to drive them in the direction they want, but I prefer to think that it is not exactly the case). If somebody can convincingly challenge that, I would be quite interested.

Apologies to everybody for switching this discussion to a sideline.

Answer 3

Has this crisis impacted (pure) mathematics, or do you believe that maths is mostly immune to it?

Immune to the replication problem, yes. But not immune to the attitudes which cause scientists to do unreplicable research in the first place. Some mathematicians will announce that a particular theorem has been proven, harvest the glory based on the fact that they have proved things in the past, and then never publish their results. Rota's Conjecture is one notorious example. Now we are in a situation where (a) nobody knows whether it is true and (b) nobody has worked on it for seven years, and probably (if it turns out that no proof actually exists) will not work on it for at least another decade.

How should results in mathematics be reproduced?

In science, it would be ideal if people dedicated research time to replicating published experimental results. This doesn't happen much because there is no glory to be gained by doing it.

The analogue in mathematics would be for people to publish new proofs of existing results, or expositions of existing proofs, which is happily much more common. I don't mean copying out well-known results in new language (Tom Leinster, The bijection between projective indecomposable and simple modules), I mean expository papers like this (Cao and Zhu, A complete proof of the Poincaré and geometrization conjectures, Asian J. Math. 10 (2006) pp. 165–492).

Even more noble are the people using proof assistant software to verify existing mathematics.

How can we expect that increasingly complicated proofs are replicated when so few people can understand them in the first place?

I think our best hope is proof assistant software. Perhaps by the end of this century, we will he living in a world where no mathematician can replicate any reasonably cutting-edge proof, yet research is still happily chugging along.