Suppose one has written a math/computer science paper that is more focused in the math part of it. I had a very complicated function and needed to find its maximum, so I used Mathematica (Wolfram) to do it. How do I explain that? "Using wolfram we find the maximum of $f$ to be $1.0328...$ therefore...". It looks very sloppy.
$\begingroup$
$\endgroup$
9
-
36$\begingroup$ If that's how you did it, that's how you explain it. Of course, the non-believers in the capability of Wolfram Mathematica (like myself) will not consider the proof complete until they verify the statement by alternative means, but the "sloppiness" is not in the explanation but in the approach itself and you won't be able to eliminate it no matter what nice words you say. So, just be straight, concise, and to the point and let the others judge whether such computations are admissible or not for themselves. $\endgroup$– fedjaCommented May 16, 2019 at 20:33
-
4$\begingroup$ I would use Mathematica as more of a guide than rely on it entirely. The maximum that Mathematica gave to you is probably very accurate; as such, depending on the precision you need, you might try finding ways to bound your function above by $2$ (or $1.033$, etc.). $\endgroup$– ClaytonCommented May 17, 2019 at 0:50
-
5$\begingroup$ Related: Computer algebra errors: mathoverflow.net/questions/11517/computer-algebra-errors $\endgroup$– joroCommented May 17, 2019 at 9:53
-
1$\begingroup$ @Clayton : If the command was NMaximize[], it is not infrequent that the supposedly near-optimal solutions it suggests are not even in the set of all feasible solutions. So, I don't think NMaximize[] can be trusted -- in contrast with Maximize[]. $\endgroup$– Iosif PinelisCommented May 17, 2019 at 13:42
-
9$\begingroup$ To add to @fedja 's comment, one thing to do is to make your code available so that other people can check it. You can publish it as a Mathematica notebook or make it available for download on a website. That way, people can check what exactly you are doing and try to verify things using alternative methods. $\endgroup$– Gabe KCommented May 17, 2019 at 15:59
|
Show 4 more comments
8 Answers
$\begingroup$
$\endgroup$
27
Welcome to MO! I believe the answer to your question depends on what Mathematica command(s) you used to find the maximum.
If you used the command Maximize[], then its output is exact and, in my view, can in general be trusted no less than the work of about any human. A reason for this belief is that Maximize[] has probably been tested at least hundreds or thousands as many times as an average proof by an average mathematician.
Still, others may disagree with this comparison between human work and the outputs of commands such as Maximize[], and so, then you may have to try to obtain the maximum in a way that can be verified by hand. Also, of course, Maximize[] can only solve comparatively easy maximization problems.
On the other hand, if you just used the command NMaximize[] -- which tries to find the maximum numerically, then its output can only be considered a suggestion -- certainly not a proof.
-
1$\begingroup$ @FrançoisBrunault : There is a command Interval[{a1,b1},{a2,b2},...] (giving the union of the intervals), and these interval objects can be the values of arguments of a function. This way, interval arithmetic is implemented. $\endgroup$ Commented May 16, 2019 at 23:35
-
21$\begingroup$ I’m one of those who would disagree that Mathematica can be “trusted no less than the work of about any human”. Mathematica absolutely has bugs, sometimes with embarrassingly simple inputs. I would at minimum cross-check with Sage/Maple/etc., if not actually verify mathematically. $\endgroup$– wcharginCommented May 17, 2019 at 2:42
-
1$\begingroup$ According to the documentation reference.wolfram.com/language/guide/IntervalArithmetic.html Mathematica has interval arithmetic only for the basic functions. $\endgroup$ Commented May 17, 2019 at 6:34
-
11$\begingroup$ Previous command continued: (ii) To make a proper comparison between the work of humans and the output of Maximize[], you may want to recall that practically all mathematicians have made mistakes as well, sometimes in embarassingly simple situations, too. $\endgroup$ Commented May 17, 2019 at 13:33
-
2$\begingroup$ In the beginning of my "Previous command continued:" comment, I made an obvious and embarrassing mistake. Of course, I wanted to say "Previous comment continued:". :-) I guess I was tired talking about Mathematica commands. $\endgroup$ Commented May 17, 2019 at 15:25
$\begingroup$
$\endgroup$
45
As is evident from the responses, different mathematicians have different attitudes towards computer calculations. If the computer calculation is not actually logically necessary for your proof, then it should be fine to say that you used Mathematica. Unfortunately (or fortunately, depending on your point of view), the world is rapidly progressing to the point where computers are actually needed to prove certain things. If a particular step in your proof requires a computation that consumes 1000 CPU hours then there is no way anyone is going to be able to "do it by hand." It doesn't sound like your computation was that large, but the point is that one is not always going to be able to sidestep the computer, and so the mathematical community needs to come up with a satisfactory approach to this sort of thing.
One approach, addressed in another MO question, is to archive or publish the computer software along with a paper. You might find some of the answers to that MO question useful. If you go this route, then I would recommend trying to perform the computation using some open-source software so that there is no problem with making the entire code publicly available.
Making the software available only addresses the issue of making your computation reproducible, and does not address the question of whether the computation is reliable. Again, assuming that the computer calculation is ineliminable, then formalizing the computation using a proof assistant (Coq, HOL Light, Lean, etc.) is the gold standard nowadays. Unfortunately, proof assistants are still not very user-friendly by the standards of the average mathematician. If you don't want to bite that bullet then probably the best you can do is to perform the calculations two or three times independently, using independent software and (preferably) a different person doing the programming. For a numerical calculation, as others have mentioned, it is important to use interval arithmetic or some other algorithm that guarantees correctness (assuming of course that the algorithm is implemented and executed correctly!). Floating-point computations are usually insufficient, according to commonly accepted standards of mathematical rigor.
-
13$\begingroup$ I’d add that using open-source software is akin to publishing in an open access venue, while using e.g. Mathematica is akin to publishing something behind a permanent paywall. $\endgroup$ Commented May 17, 2019 at 18:40
-
5$\begingroup$ what is the balance of publishing in Elsevier journal, compared to publishing in a refereed arxiv.org-based journal? well, these without funds wont’t read you. Oh, but the Elsevier shareholders obviously would be happy... $\endgroup$ Commented May 17, 2019 at 19:20
-
3$\begingroup$ also, imagine for a moment Wolfram Inc stopping with Mathematica for some reason. then all this code would quickly become next to useless... just like texts written in ChiWriter... $\endgroup$ Commented May 17, 2019 at 19:24
-
4$\begingroup$ @DimaPasechnik : I don't see much relevance of the Elsevier/Springer/etc matter here. Also, in the event that Wolfram Inc stops Mathematica, there still will be a solution: to check the claims by using other software. However, I have rather seen cases when OSS gets abandoned. (As a practical matter, except for the developers, I think hardly anyone would ever check any part of the computer code itself, be it Mathematica or OSS). $\endgroup$ Commented May 17, 2019 at 22:11
-
1$\begingroup$ @IosifPinelis : If you read through the comments here mathoverflow.net/a/316177/3106 you will see that docker is at least a partial solution to the problem of the operating system being abandoned. $\endgroup$ Commented May 17, 2019 at 23:28
$\begingroup$
$\endgroup$
The answer to this will differ from one field of math to another. In my opinion (I work primarily in combinatorics and algebraic geometry) a paper should contain enough data that the reader could verify its claims without relying on a particular software package, or should link to a repository of such data.
The main paper where this came up for me is Version 1 of my paper with Robert Kleinberg and Will Sawin. In Section 7 of this paper, we need to verify that a certain linear program, whose constant terms are polynomials in an algebraic number $\rho$, is feasable. What we do in the paper is to find a solution point whose exact coordinates are polynomials in $\rho$ and provide the reader with both those exact polynomials and with their numeric values to enough accuracy to confirm that it is a solution. The skeptical reader may use the exact formulas to verify that the equality conditions hold and may then compute the numerical values on their own to enough accuracy to check that the inequalities hold. We didn't provide the (quite hackish) Mathematica code which computed this witness data.
We printed the first few solutions in the paper and then uploaded a text file to the arXiv (see "Download Source" here) with more.
This material is missing from the final version of the paper, because a more general result was proved by Pebody and Norin. Otherwise we would have done something similar in the final paper; I don't know whether the journal would have wanted all the data in print, would have wanted to host the extra data themself, or just to keep it on the arXiv, but I would want to do one of these.
$\begingroup$
$\endgroup$
14
25 years ago I published a paper where there were computer-based enumerations of certain combinatorial objects. Unfortunately I did not publish the code (in GAP, an OSS system) and it vanished after a place where I used to be a postdoc some years later shut down an ftp server with my files). Fortunately some results there were then verified by hand. Just a month ago I had to sit down and re-do these computations for the remaining cases, as someone asked for it. Fortunately it did not took too much time, and I even learned something new (and publishable).
Anyhow, publish your code, and make it sufficiently clean to be understood years later...
-
$\begingroup$ I think we need to distinguish "computer aided proof" (your case) and "computer calculation" (OP's case). I support the former and regard it as a solid math proof, but I strongly oppose the latter as proof in any sense. $\endgroup$– Hao ChenCommented May 17, 2019 at 19:58
-
$\begingroup$ well, it is a computer calculation, what I did, using a relatively straightforward recursive search sped up by symmetry considerations (computing orbits). $\endgroup$ Commented May 17, 2019 at 20:01
-
1$\begingroup$ I also have papers where result depends on running numerical optimization - the answer however verifiable by cheap checking of optimality conditions (which basically only needs matrix arithmetic) $\endgroup$ Commented May 17, 2019 at 20:04
-
3$\begingroup$ I use "computer aided proof" for the case where a human set up a plan and computers do the work. This can then be verified by checking the plan and the implementation. I use "computer calculation" for the case where a human use a software, whose algorithm he does not read / not completely understand, just for an output. $\endgroup$– Hao ChenCommented May 17, 2019 at 20:06
-
1$\begingroup$ @TimothyChow my requirements do not extend to compiler, OS or hardware, but let me explain. Fot a proof to be acceptable, there should be a purely logic possibility that any human being, given infinitely much time and enough knowledge, could work out every step of the algorithm. This is possible once the algorithm and the necessary knowledge is accessible. OS etc. doesn't matter here. But if aliens come with a machine that is said to do multiplication with a more advanced method that humans do not have access/understand, then mathematicians should not use this machine for any proof. $\endgroup$– Hao ChenCommented May 18, 2019 at 4:22
$\begingroup$
$\endgroup$
3
Other answers and comments have made passing suggestions that you could perhaps consider making your code available. I think that that is far too weak: it is imperative that you should make your code available, and I would not accept your paper as an editor or referee if you did not do that. You should upload the code to the arxiv as a supplementary file, because that is a fairly reliable way to make sure that the original version remains available. You could also put it on github or a personal web page.
-
2$\begingroup$ Personal web pages change frequently. Something more stable such as the arXiv seems much more desirable. $\endgroup$ Commented May 18, 2019 at 18:08
-
1$\begingroup$ @AndrésE.Caicedo Are we reading the same thing? It sounds like Neil Strickland is suggesting putting the code on the arXiv, and "also" possibly ("could") on a personal webpage/github. Presumably for ease of access – getting the source on arXiv is a bit annoying. $\endgroup$ Commented May 20, 2019 at 12:32
-
$\begingroup$ To clarify, I am suggesting that code could go on a personal web page as well as the arxiv, it would certainly not be satisfactory to have it only on a personal web page. $\endgroup$ Commented May 20, 2019 at 14:54
$\begingroup$
$\endgroup$
How do I explain that? "Using wolfram we find the maximum of $f$ to be $1.0328...$ therefore...". It looks very sloppy.
Well, it looks sloppy because it kind of is sloppy by your description. Maybe the right framework for thinking about your question is to imagine that the maximization you are claiming is a component in a claimed proof of, say, the Riemann hypothesis. Would you pay serious attention to a paper containing such a claimed proof that purported to rely on an opaque software-based maximization without providing any code, let alone any guarantees that the code and the underlying software platform it runs on do what they say they do? I wouldn’t, nor do I expect any serious person to allow such a paper to be published.
Now, I‘m guessing the problem your paper claims to solve isn’t as important as the Riemann hypothesis. The question of what authors of math papers can get away with and still get their paper published is distinct from the question of what authors should do as an ideal to aspire to that would make their paper’s claims truly convincing and watertight. People do get away with all kinds of minor (or even major) sloppiness, of the kinds you mention and of other kinds having nothing to do with the use of computers, all the time. I know I have. Nonetheless, since you seem to be asking about what is the “right” way to explain your result, the answer (assuming you are publishing what is meant to be a rigorous theorem with a proof that’s up to the standards of a good pure math journal) is: provide as many details about your claimed maximization as are needed to convince almost everyone in the research community of the validity of your claims.
For myself, seeing your Mathematica code would be an absolute minimum to satisfy this condition. Depending on the precise nature of your calculations, I may also want to see that they satisfy some combination of the following conditions:
- They are based on algorithms that have been around for a long time and everyone is sure are correct.
- They are based on algorithms that have themselves been published in peer-reviewed journals.
- They can be replicated with relative ease in software packages other than Mathematica.
Hope this helps.
$\begingroup$
$\endgroup$
What you did is a numerical investigation, so make a numerical claim, nothing more. It should not be claimed as a proposition, or be used as a step in a proof/argument. In particular, I don't like the "therefore" in your example. No matter how many times a numerical method has been tested, it is never a proof and should not be considered solid in the mathematical sense (except for, maybe, those symbolic or analytic result that can be verified by hand).
In many cases of mathematics, the concrete number does not matter. You might be able to prove the existence of a maximum within a range (could be very hard). This is usually better (trustworthy and verifiable) than an actual number. In fact, we often need to prove the existence of a solution whose value has been known since long (recently been there ...)
$\begingroup$
$\endgroup$
6
Did you try softwares doing rigorous numerical computation? For example arb is a C library freely available in Sage. Such software gives guaranteed enclosures $f(I) \subset J$ where $I,J$ are intervals. So an obvious (but not necessarily optimal) algorithm for finding the maximum of your function $f:I \to \mathbb{R}$ is to subdivide your interval $I$ and apply $f$ to each subinterval, and keep subdividing until you get the maximal value to the precision you want. Personally I would judge this as rigorous and acceptable in a proof. This should work as long as the functions you need are present in arb, at worst the algorithm will be too slow.
-
$\begingroup$ I don't see how this is relevant. $\endgroup$ Commented May 20, 2019 at 12:33
-
$\begingroup$ @NajibIdrissi The OP did a computation using Mathematica, which is not rigorous in the usual mathematical sense (I'm not saying this function may have bugs, just that he's trying to maximize a function, so Mathematica uses floating-point numbers, which may cause some problems, see e.g. the first integral here: fredrikj.net/blog/2017/11/… ). Thus a natural thing to do is to use a software implementing floating point computations with rigorous bounds (this is usually called interval arithmetic). $\endgroup$ Commented May 20, 2019 at 12:40
-
1$\begingroup$ Mathematica doc: "If Maximize is given an expression containing approximate numbers, it automatically calls NMaximize". $\endgroup$ Commented May 20, 2019 at 12:45
-
$\begingroup$ Sure, but that's not the question that was asked. The question was how to present results obtained (no matter the method) in a paper. I'm not sure why everyone jumped straight on the NMaximize bandwagon. $\endgroup$ Commented May 20, 2019 at 13:40
-
1$\begingroup$ @NajibIdrissi If there is no way to present it in a paper because it is not rigorous enough (no matter the presentation), then I think this is a solution. At least this is my interpretation of the question that it was about presenting but also making things rigorous. $\endgroup$ Commented May 20, 2019 at 13:42