Do empirical studies have a place in contemporary mathematics research?

Question

I was thinking about the Collatz conjecture a while back (I know, not the healthiest thing to think about). It occurred to me that while I might not be able to prove it true for all positive integers, I could spend 20 minutes and script up a simulation to generate a few million pseudorandom integers and test them to see if they follow the conjecture, and then publish my results as an empirical study such as is done every day in the physical and social sciences:

We tested ten million positive integers generated according to the pseudorandom algorithm defined by Jones [cite], then ran them against our TestCollatz(EnormousInt) function. We observed that 100% of the sample generated met the conjecture with a 99% confidence interval of plus or minus 0.1%. Conclusion: The evidence strongly supports the Collatz conjecture. We hope that future research in this area will be able to test billions or even trillions of integers to gain even more confidence in the accuracy of Collatz's model.

I soon realized that there was no algorithm to generate a statistically representative sample across a set of cardinality $\aleph_0$ with finite computing resources, and so my study idea was fruitless.

Stepping back from my misguided thought experiment, I realized I couldn't recall hearing of a recent empirical study in mathematics. Rather, the focus is on formal proofs and 100% confidence solely through logical and mathematical reasoning (not experimental data). Does empirical research exist or happen? I can find plenty of empirical research in mathematics education (e.g. checking whether starting children on arithmetic at age five rather than six raises standardized test scores), but mathematics education research is really a social science and not mathematics per se.

If there are recent empirical papers in mathematics, can you give me some sample citations of some interesting or notable ones to read? If empirical research is not done in mathematics today, why?

To be clear, I'm not asking about the Collatz conjecture specifically or why my idea was nonsense. I already know why. I'm asking if there are good examples of empirical studies in research-level math.

In terms of what I mean by "empirical study", I mean the standard "scientific method" of doing a literature review, forming a hypothesis, designing a study, randomizing experimental and control groups, doing the study, gathering the data, statistically analyzing it, forming a conclusion, and publishing. For example, if a medical researcher wants to show that a new drug works, they don't publish a five-page proof consisting of chemical reaction equations that they claim the drug triggers in the body, but rather they actually give the drug to a sample group and see what happens.

Answer 1

You can browse the journal Experimental mathematics, which publishes original papers featuring formal results inspired by experimentation, conjectures suggested by experiments, and data supporting significant hypotheses.

Answer 2

I would say an emphatic yes! Personally, I study representation theory and cohomology of finite groups. In order to get some idea of what's going on before developing theory or before formulating conjectures, I tend to run a large number of experiments using Magma, on examples in a range where this is possible. In cases where this produces lists of numbers, I use Sloane's online handbook of integer sequences to identify what might be going on. A good example is my recent paper with Pavel Etingof, "On cohomology in symmetric tensor categories in prime characteristic." Here, we give a conjecture for the cohomology of certain new symmetric tensor categories in prime characteristic, and prove that this conjecture is true up to "F-isomorphism". The evidence was literally many hundreds of computations using Magma, and the pattern that emerged was very compelling.

Answer 3

I don't know if mathematicians consider cryptography a "true" area of math, but it is at the minimum a fairly theoretical applied area of math that mixes

• a strong emphasis on provable results/techniques, and
• a strong emphasis on experimental results/techniques.

Ideally, all results would be provable, and cryptographers would develop schemes that are unconditionally secure in certain models of computation. This runs into issues though --- proving impossibility results on efficient computation is notoriously difficult (for example $P$ vs $NP$, it suffices to prove any super-polynomial lower bound on computing SAT. I believe the best lower bounds we have are of the form $6n$ in some explicit TM model of computation --- i.e. slightly better than the trivial $n$ lower bound, but still $\Omega(n)$).

Given this issue, cryptography often factors into two parts

• hardness assumptions, which may have some provable results, but are mostly validated via experimental means, and
• constructions, which are formally proved secure, provided the underlying hardness assumption is secure.

While one can theoretically examine hardness assumptions (for example, showing things called "randomized self reductions" which imply that average-case instances of the underlying problem are roughly as hard as the worst-case instances), all it takes to invalidate a hardness assumption is one good experimental paper. For example

• the signature scheme RAINBOW (in the NIST competition for a "post-quantum" signature) was attacked last year in the paper Breaking RAINBOW takes a weekend on a laptop. There is of course a theoretical description of the attack, but provable results regarding the attack don't really matter when one can experimentally verify that it works. Similarly

• cryptosystems based on the Super-Singular Isogeny Diffie Hellman problem suffered devestating attacks last year. Again, while one can provably bound the effectiveness of the various algorithms, this is really a second-order concern to the fact that someone can write some SAGE code that extracts the secret key out of the cryptosystem.

There are other experimental trends in the field, for example the LLL basis reduction algorithm is known to perform "better on average/in practice" than its worst-case bounds would suggest. There has been much work (motivated by the potential for cryptanalysis) experimentally trying to understand precisely what this means. See for example Nguyen and Stehle's 2006 work LLL on average, though there are many works similar to this. This seems really most closely to what you're asking about, and it is widely-cited for its experimental results in the field.

Answer 4

While there are projects that check more and more cases of some conjectures, it is rare that they really increase the degree of certainty that these conjectures are indeed true. 10 more (or 10^10 more) numbers for which Collatz conjecture holds are unlikely to change anyone's views on this.

What empirical studies are used for is figuring out what the problem at hand is, that is, to form conjectures (rather then prove them!). The most spectacular recent example is of course this Nature paper, which uses machine learning to establish a pattern between some object in knot theory and representation theory (the observed results are proved in another paper!).

Another type of result which can be considered empirical are of the following sort: prove a conjecture for all but finitely many cases (with bound on what they are) and verify those remaining cases on a computer. If I recall correctly, this is how ternary Goldbach conjecture was proved by Helfgott (see this paper for the empirical part).

Answer 5

coudy's answer, pointing you to the Experimental Mathematics, is the right answer, and there are already other MO questions about experimental mathematics that are relevant, but I can't resist giving some examples.

• Here's one example that has a similar flavor to yours, but with the Riemann hypothesis instead of the Collatz conjecture. In his Math. Comp. paper, On the distribution of spacings between zeros of the zeta function, Andrew Odlyzko reported on a computational study of Montgomery's pair correlation conjecture. There are of course many other examples in Math. Comp.; often, the criterion for publication here is some kind of algorithmic novelty (e.g., new algorithmic techniques had to be devised in order to push the computation out much further than before).

• A workhorse of experimental mathematics is the empirical discovery of relationships between real numbers by applying lattice basis reduction algorithms to their decimal expansions. Some of these may be proved, while others remain conjectural. A sample paper that I like is About a New Kind of Ramanujan-Type Series, by Jesús Guillera, which among other things presents Gourevitch's conjecture: $$\sum_{n=0}^\infty \frac{1+14n+76n^2+168n^3}{2^{20n}}\binom{2n}{n}^7 = \frac{32}{\pi^3}.$$

• Siemion Fajtlowicz's paper On Conjectures of Graffiti reported on a program that searched empirically for graph-theoretic conjectures. Of course, most were either trivial or false or known, but some turned out to be new conjectures which were not trivial to prove.

Answer 6

The Birch and Swinnerton-Dyer conjecture was formulated on the basis of substantial computer calculations. They looked at the growth of products $\prod_{p \leq x} N_p(E)/p$ for large $x$ and various elliptic curves $E$, and found patterns distinguishing different ranks. Discussions with Cassels, it seems, directed them towards a reformulation of their work in terms of the order of vanishing of a the $L$-function of $E$ at the point $1$ (which, for the elliptic curves they were studying, could be numerically approximated even if the general case took $30+$ more years to be justified). The introduction to their second paper here discusses their investigations.

Answer 7

I've spent a lot of time in recent years thinking about the relational complexity of finite permutation groups. For a permutation group $G$, the relational complexity of $G$ is an integer, at least $2$ which is a bit tricky to define. It is also quite tricky to compute, even for well-known families of permutation groups.

For a long time the big problem concerning relational complexity was the classification of the primitive finite permutation groups with relational complexity equal to $2$. The form of this classification was conjectured by Cherlin [1]. My understanding is that Cherlin arrived at his conjecture after doing a great deal of empirical research -- computations with lots and lots of different primitive groups, attempting to see patterns etc.

In my own work on relational complexity I have done similarly. Moreover, it seems to me that the Classification of Finite Simple Groups has greatly increased the efficacy of an "empirical approach" when it comes to questions concerning finite (permutation) groups: typically one conceives of a possible theorem that may be approached via some kind of "reduction to simplicity" -- you show that if the result is true for (almost) simple groups, then it is true in general. After this reduction one can apply CFSG and work through the simple groups family by family.

Keeping this in mind, then, I often end up messing about with "typical" simple groups to see if my imagined result might be true, possibly running a bunch of computations on a suitable sample of these typical groups, and testing to see if my intuition is correct. Usually it is isn't correct first time around, but I might be able to tweak my ideas and eventually arrive at some statement that the computer isn't immediately telling me is wrong.

Perhaps this empirical approach differs a little from the scenario you describe because it doesn't tend to have the randomised element that you describe. What is more, the process I'm describing does not normally form the content of a paper -- it is more often part of the preliminary work that tells me whether or not a theoretical result might be true / possible.

[1] Cherlin, Gregory, Sporadic homogeneous structures, Gelfand, I. M. et al., The Gelfand Mathematical Seminars, 1996-1999. Dedicated to the memory of Chih-Han Sah. Boston, MA: Birkhäuser. 15-48 (2000). ZBL0955.03040.