61
$\begingroup$

Lately, I have been learning about the replication crisis, see How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth (good YouTube video) — by Michael Shermer and Stuart Ritchie. According to Wikipedia, the replication crisis (also known as the replicability crisis or reproducibility crisis) is

an ongoing methodological crisis in which it has been found that many scientific studies are difficult or impossible to replicate or reproduce. The replication crisis affects the social sciences and medicine most severely.

Has the replication crisis impacted (pure) mathematics, or is mathematics unaffected? How should results in mathematics be reproduced? How can complicated proofs be replicated, given that so few people are able to understand them to begin with?

$\endgroup$
69
$\begingroup$

Mathematics does have its own version of the replicability problem, but for various reasons it is not as severe as in some scientific literature.

A good example is the classification of finite simple groups - this was a monumental achievement (mostly) completed in the 1980's, spanning tens of thousands of pages written by dozens of authors. But over the past 20 years there has been significant ongoing effort undertaken by Gorenstein, Lyons, Solomon, and others to consolidate the proof in one place. This is partially to simplify and iron out kinks in the proof, but also out of a very real concern that the proof will be lost as experts retire and the field attracts fewer and fewer new researchers. This is one replicability issue in mathematics: some bodies of mathematical knowledge slide into folklore or arcana unless there is a concerted effort by the next generation to organize and preserve them.

Another example is the ongoing saga of Mochizuki's proposed proof of the abc conjecture. The proof involve thousands of pages of work that remains obscure to all but a few, and there remains serious disagreement over whether the argument is correct. There are numerous other examples where important results are called into question because few experts spend the time and energy necessary to carefully work through difficult foundational theory - symplectic geometry provides another recent example.

Why do I think these issues are not as big of a problem for mathematics as analogous issues in the sciences?

  1. Negative results: If you set out to solve an important mathematical problem but instead discover a disproof or counterexample, this is often just as highly valued as a proof. This provides a check against the perverse incentives which motivate some empirical researchers to stretch their evidence for the sake of getting a publication.
  2. Interconnectedness: Most mathematical research is part of an ecosystem of similar results about similar objects, and in an area with enough activity it is difficult for inconsistencies to develop and persist unnoticed.
  3. Generalization: Whenever there is a major mathematical breakthrough it is normally followed by a flurry of activity to extend it and solve other related problems. This entails not just replicating the breakthrough but clarifying it and probing its limits - a good example of this is all the work in the Langlands program which extends and clarifies Wiles' work on the modularity theorem.
  4. Purity: social science and psychology research is hard because the results of an experiment depend on norms and empirical circumstances which can change significantly over time - for instance, many studies about media consumption before the 90's were rendered almost irrelevant by the internet. The foundations of an area of mathematics can change, but the logical correctness of a mathematical argument can't (more or less).
$\endgroup$
  • 8
    $\begingroup$ @QuartoBendir I think 2 and 3 provide some check against this. Modern algebraic geometry is hard to learn, but a modern student who has taken the time to learn it properly can come up with very simple proofs of results that would have been considered very hard a century ago. I think most areas of math gradually get easier over time as people discover new connections and generalizations that allow future readers to get more bang for their buck. $\endgroup$ – Paul Siegel Sep 5 at 6:07
  • 10
    $\begingroup$ @QuartoBendir By my understanding the issues that have reached crisis levels in the sciences are twofold: first, that some classical, textbook studies (e.g. the Stanford prison experiment) turned out to be fraudulent or wrong; and second, that very few results are ever replicated at all, even by experts in the same field. I do think that the practice of experts applying and generalizing each other's results helps mathematics avoid these problems even in the shorter term, though admittedly it doesn't do much for mathematicians in other fields. $\endgroup$ – Paul Siegel Sep 5 at 6:35
  • 11
    $\begingroup$ Why do you phrase it as if Mochizuki has a proof but only some experts disagree? Isn't it the case that most experts do not believe he has a proof, and only those close to Mochizuki believe that he does? In fact, the very fact that he can publish it in a journal that has himself as chief editor shows without doubt that mathematics will have a replication crisis (however small) if mathematicians do not maintain a list of reputable journals. $\endgroup$ – user21820 Sep 6 at 2:58
  • 7
    $\begingroup$ As far am I am aware the only high-profile person from the Anglo-European world who believes in Mochizuki is Fesenko. $\endgroup$ – user164740 Sep 6 at 9:41
  • 7
    $\begingroup$ @user21820 You are reading intent here where none is present. The phrase "dedicated experts" in the original answer refers to both the critics and advocates of the work, and the term "foundational" in context refers to the role of an idea in an argument, not its correctness. I'm going to leave your edits as a show of good faith (and I genuinely don't care), but in exchange I'd like you to go grind your ax somewhere else - none of this discussion is relevant to the question or the answer, and it's pretty clear that only one of us is actually neutral toward the irrelevant dispute. $\endgroup$ – Paul Siegel Sep 6 at 13:52
15
$\begingroup$

How can we expect that increasingly complicated proofs are replicated when so few people can understand them in the first place?

My answer to that is that we do not expect them to be replicated in the usual sense of this word (repeated and included into textbooks with just minor cosmetic and stylistic changes). Rather we expect them to be gradually simplified and streamlined either through changing the proofs themselves by finding a shortcut or replacing the whole argument with a completely different one, or by building a theory that is locally trivial but proceeds in the direction of making the proof understandable and verifiable much faster than the currently existing one. The latter is exactly what Mochizuki tried to do though his goal was rather to just reduce the difficulty from "totally impossible" to "barely feasible" and the prevailing opinion is that he failed in the case of the ABC conjecture though he has succeeded in several other problems.

The first approach is more common in analysis (broadly understood), the second is more common in algebra (also broadly understood), but you can try to play either game in either field. My own perception of what is proved and what is not borders on solipsism: I accept the fact as proven if I've read and understood the whole argument or figured it out myself. So most mathematics remains "unproved" to me and, apparently, will stay unproved for the rest of my life. Of course, it doesn't mean that I'm running around questioning the validity of the corresponding theorems. What it means is that I just never allow myself to rely in my own papers on anything that I haven't fully verified to my satisfaction, try to make my papers as self-contained as possible within practical limits, and that I consider the activity of simplifying the existing proofs as meaningful as solving open questions even in the case when the proofs are reasonably well-known and can already be classified as "accessible". But not everybody works this way. Many people are completely happy to drop a nuke any time they have an opportunity to do it and there is nothing formally wrong with that: the underlying point of view is that our time is short, we have to figure out as many things as possible, and the simplifications, etc. will come later. Probably, we need a mixture of both types to proceed as efficiently as we can.

So I would say that the mathematics is reasonably immune to this crisis in the sense that mathematicians are aware of the associated risks, take them willingly, and try to gradually build the safe ground of general accessibility under everything though the process of this building is always behind the process of the mathematical discovery itself. The same applies to physics and medicine though the gap between the "front line" and the "safe ground" there may be wider. In fact, it applies to any science that deserves to be called by that name. As to the so called "social sciences", they are often done at the level of alchemy and astrology today in my humble opinion (and not only mine: read the Richard Feinman critiques, for example) but we should not forget that those were the precursors to such well-respected sciences as chemistry and astronomy/cosmology, so I view the current crisis there as a part of the normal healthy process of transitioning from the prevailing general "blahblahblah" and weathervane behavior with respect to political winds to something more substantial.

Edit: Paul Siegel has convinced me that things have indeed changed since the time I took (obligatory) courses of Marxist philosophy and the history of communist party, though this change may be not easily visible to the general public because it mainly happens outside academia and is driven primarily by company business interests, so a huge part of it occurs behind closed doors (Paul, please correct me if I misinterpreted what you said in any way). So my statement that the current social sciences are not capable of something beyond general blahblahblah is no longer valid and I retract it. However I still maintain the opinion that it is blahblahblah rather than hard data analysis or other scientific approach that drives many public political and social discussions and decisions of today (I don't know what happens here behind the closed doors, of course, and it may be that, like in advertising, what we see is just what shepherds choose to show to their sheep to drive them in the direction they want, but I prefer to think that it is not exactly the case). If somebody can convincingly challenge that, I would be quite interested.

Apologies to everybody for switching this discussion to a sideline.

$\endgroup$
  • 7
    $\begingroup$ I think the second half of this comment is typical of the low quality of commentary when it comes to the social sciences, which are a broad gamut of fields, all of which are different. Richard Feynman's thoughts on the topic 70 years ago are not the final word on the subject. There is a replication crisis in social psychology, which is not even all of psychology. Is it a general problem across all of social science? Who knows? But mention the word "social" and suddenly everyone's an expert. $\endgroup$ – arsmath Sep 6 at 14:56
  • 3
    $\begingroup$ @arsmath Agreed. I haven't explained myself well enough preferring to curtail the subject to a few words and to refer to the opinion of a way more eloquent and knowledgeable person. If you want a more expanded version of what I think, I can provide it later. General psychology, by the way, is a part of medicine in my classification. We just understand the words "social sciences" slightly differently, that's mainly it. Upvoting your comment but changing nothing in the main text :-) $\endgroup$ – fedja Sep 6 at 17:18
  • 5
    $\begingroup$ I suspect your impression of what modern social science looks like is a couple decades out of date. The level of scientific rigor found in digital advertising, market research segmentation, forecasting consumer sentiment, and so forth would meet whatever standard you care to throw at it - there are billions of dollars riding on tiny fractions of a percent accuracy. The problem used to be "How accurately can we model human behavior?" The question has become "Is it ethical to model human behavior as accurately as we can?" $\endgroup$ – Paul Siegel Sep 6 at 23:27
  • 6
    $\begingroup$ @D.S.Lipham A lot of the best social science research isn't happening in universities anymore - this is because the underlying datasets are huge and proprietary. Think Amazon's consumer demand dataset, Google's clickstream data, Facebook's social graph, etc. Academic social science researchers generally work with different problems, smaller budgets, and less accountability, but a number of specific disciplines are thriving - computational linguistics, for instance, has made epoch-defining breakthroughs in the last year or two. $\endgroup$ – Paul Siegel Sep 7 at 0:04
  • 3
    $\begingroup$ @PaulSiegel OK, then it is a mixture of the progress and the takeover in my opinion :-). That's good actually. Probably the best option of the three. I'll update my "outdated opinion" accordingly ;-) I hope that the ethical problems you are talking about will be also decided by a mixed team like yours, not by the people who are telling us what to think and how to behave today. Thanks for your contribution to this conversation! $\endgroup$ – fedja Sep 7 at 4:29
14
$\begingroup$

Has this crisis impacted (pure) mathematics, or do you believe that maths is mostly immune to it?

Immune to the replication problem, yes. But not immune to the attitudes which cause scientists to do unreplicable research in the first place. Some mathematicians will announce that a particular theorem has been proven, harvest the glory based on the fact that they have proved things in the past, and then never publish their results. Rota's Conjecture is one notorious example. Now we are in a situation where (a) nobody knows whether it is true and (b) nobody has worked on it for seven years, and probably (if it turns out that no proof actually exists) will not work on it for at least another decade.

How should results in mathematics be reproduced?

In science, it would be ideal if people dedicated research time to replicating published experimental results. This doesn't happen much because there is no glory to be gained by doing it.

The analogue in mathematics would be for people to publish new proofs of existing results, or expositions of existing proofs, which is happily much more common. I don't mean copying out well-known results in new language (Tom Leinster, The bijection between projective indecomposable and simple modules), I mean expository papers like this (Cao and Zhu, A complete proof of the Poincaré and geometrization conjectures, Asian J. Math. 10 (2006) pp. 165–492).

Even more noble are the people using proof assistant software to verify existing mathematics.

How can we expect that increasingly complicated proofs are replicated when so few people can understand them in the first place?

I think our best hope is proof assistant software. Perhaps by the end of this century, we will he living in a world where no mathematician can replicate any reasonably cutting-edge proof, yet research is still happily chugging along.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.