# How to distribute the source of programs used in a paper?

I have written a paper, which includes an appendix discussing how to obtain numerical evidence for the result of the paper. Now the computation essentially works as follows:

• Create a large tridiagonal matrix.
• Compute its eigenvalues.
• Compute the difference of consecutive eigenvalues, and output it.

The implementation of such an algorithm is rather straightforward, but in order to look at large matrices, I started using algorithms from a package called LAPACK, which turned out to be faster then regular algorithms provided by Matlab. (I'm no specialist, so not exactly sure what happens).

I am curious if one should provide the source code for such a computation, and if yes in what form. I cam up with the following options:

• Pseudocode (as above)
• Simplified matlab, that works with any installation of matlab, but is too slow to actually do the computations.
• The real code, which most people will not be able to get to run without some effort.

I am also curious if one should include some sort of source code in the paper, and if yes, in what form? Or what people have done in such a case...

The simplified code is available at: http://math.rice.edu/~hk7/ftp/matlab_code/SkewSpecDense.m

I have not put the real code online, because it requires external packages, and I am not sure how easy it is to install them...

-
I think answers tend to be journal-specific. If I was in this situation I'd choose the journal based on what kinds of publishing options they have, which fits in best with how I'd like to see the paper published. Most often for pure math papers you see at most a pseudo-code sketch, and the author hosts their code on their personal web-pages. Some authors go so far as to maintain open-source repositories for larger projects that include the code relevant to their paper. – Ryan Budney Sep 17 '10 at 13:39
Mathworks note: "If S is sparse and symmetric, you can use d = eig(S) to return the eigenvalues of S. If S is sparse but not symmetric, or if you want to return the eigenvectors of S, use the function eigs instead of eig." – Steve Huntsman Sep 17 '10 at 13:48
FWIW, here's the LAPACK license: netlib.org/lapack/LICENSE – J. M. Sep 17 '10 at 14:21
@Ryan: The numerical evidence is not the huge junk of the paper. I actually prove the result, in some parameter range. The reason I want to include the code is a bonus, that the result remains valid in the other parts of the parameter range. So I want a solution that will fit prefer any math journal. – Helge Sep 17 '10 at 15:55
A similar question mathoverflow.net/questions/17344/… – j.c. Sep 17 '10 at 17:26

My preference is detailed pseudocode, at a high-enough level of abstraction to allow understanding the algorithm.

Of course, as pointed out by Ryan Budney's comment, it depends strongly on what the journal requirements are and in which journal you publish. However, I feel strongly that the complete code-set which you use should be available from some resource, either through the journal article's publsher, or from your own website, your academic website, or via Arxiv.

If the pseudo-code is detailed enough to allow reimplementing the algorithm straightforwardly by another mathematician, then that should be sufficient.

If the pseudo-code has to leave out certain details which are germane to the computation, then the interpreted code which implements the algorithm in a numerical computational package (such as Maple, Matlab, Sage, or Octave or Scilab (download link ) which are free open source software packages capable of running code similar to or equivalent to matlab) should be provided.

Why not provide both? -- If you can provide a link to your own webpage for the paper, or for its supporting supplemental materials, I don't see why you couldn't provide both the interpreted code and the compileable C or C++ code on your webpage, unless there are copyright issues involved such as if you did not write all of the code yourself and do not have the right to release all of the code source. I am a supporter of free open-source software and the Gnu organization's GPL licensing, which would allow others to benefit from your code and to contribute back to it via incremental improvements.

I suggest that you specify which version of software package, operating system, compiler, and/or library you used in running your program or in creating the binary application from your code. This is necessary because different versions of Octave (2.3 vs. 3.0) or Matlab (R10, R13, etc.) or any software package may implement or include different routines and may not be capable of correctly running your software program.

I would recommend that if particular packages are necessary in order to run the interpreted code in Octave or Matlab that you list which packages they are. In the same vein, if your C or C++ code requires particular libraries such as LAPACK or BLAS, make sure to list them in a text file or in a header file. If you know how to use the make program, you can create a makefile to help others in compiling your software.

The make program, the Gnu compiler collection, and many other development tools are all standard parts of Gnu/Linux distributions, such as Debian.

My preference is detailed pseudocode, at a high-enough level of abstraction to allow understanding the algorithm.

-
I have some objections against both: I expect most of my readers not too be too computer knowledgeable. So I would prefer to have something simple available. – Helge Sep 17 '10 at 14:26
I agree with you, as my preference is detailed pseudocode, in-line in the paper if possible, or in the appendix, with an URL pointing to detailed or complete source code available on your website or ArXiv. Most readers may not be computer knowledgeable, but those who wish to implement (to test or use) your algorithm will probably also want to use it on larger matrices and will want it to run quickly. Having access to the real source code will make that possible. – sleepless in beantown Sep 17 '10 at 15:14

You should make all three forms publicly available: pseudo-code, simplified code, and working code, with as much as possible at the arXiv or another publicly maintained site. However, I don't think permanent storage of the working code is as central an issue as it is for research articles, since computing environments evolve quickly. The most important thing to worry about is its availability for say 10 years (but the pseudo-code, more permanently).

Online storage is extraordinarily cheap. I don't see a rationale for skimping. The only consideration is organizing it so people understand what's there, and have guidance as to which version (if any) they might want. The three forms serve different purposes. Even if nobody ever downloads the full version, it only costs you or somebody pennies, and it may even reduce your time and trouble just to provide it.

Sometimes there's a computational task someone wants to perform that is just a step in a bigger project. Programming tends to consume a fair amount of time per idea, even if you are very clear on what you're doing, and programming skill and speed varies widely between different people. You should make it as easy as possible for them to make use of your work.

Sometimes people are thinking about solving related problems where the code won't be directly helpful, but the ideas may be, and sometimes people just want to check whether what you're doing is correct. Pseudocode is much preferable in a case like this.

Sometime people may want to actually use the code, but they don't have your infrastructure installed. For such people, it's good to provide a stripped-down form that doesn't require much to start with --- once they have it working in some baby form, they can add in external libraries, or perhaps optimize it in different ways. Maybe they'll even improve it.

-

Include the source code as auxiliary files when you submit to the ArXiv. That's a more permanent and safe location than your own webpage. In the actual article just put pseudo-code and a link to the ArXiv source.

If you have a lot of code that approach won't work, and you'll have to host it yourself.

See http://arxiv.org/abs/1007.1730 for an example of how Scott and I dealt with this issue, there's a rather extensive worked example so that people can figure out how to check the program's output locally on their own. There we have a huge package we've written that the program uses, so we have a server set up that anyone can download the package from.

-
How do access the code? I can't find it in the gziped file. – Helge Sep 17 '10 at 14:29
That was an example where the code was too large to fit in the source, so look at pages 14 and 15 for instructions. – Noah Snyder Sep 17 '10 at 15:27

In my opinion, you should absolutely distribute the actual source somewhere. It is important evidence for your claims. Sure, 95% of your readers will trust you that the results are as you say, but the 5% who want to check or extend your work are the ones you care about most.

Most authors I have seen put the source on their personal webpages, and provide a URL in the article. If your work is going to appear in a journal that regularly publishes computation intensive papers, you might see what their normal practice is.

-
I agree. However, the life-span of a personal webpage is much shorter than the life time of its owner. Therefore a repository for such software should be set up and sponsored. For example by using the \$1 mil. prize the Clay Institute is looking to spend for the benefit of mathematics. Let me suggest: 1/3 mil. for MO, 1/3 mil. for the arXiv and 1/3 mil. for a mathematical software repository. This would "increase and disseminate mathematical knowledge" and maybe such a solution would also please Perelman as it would acknowledge the way he shared his work with the public? – Bruce Arnold Sep 17 '10 at 16:40

One option might be to include your code in Sage: http://sagemath.org/development.html Code that gets included with Sage is peer reviewed, and there are standards of testing and documentation. The code also must work on a wide range of platforms, and continue to work on those platforms in new releases. Note that Sage includes LAPACK, but of course Sage is open source so it does not include MATLAB, so distributing with Sage may not be an option for you.

There is a directory in the main Sage library devoted to code that isn't meant to be used on a regular basis by Sage users, but is instead connected with published papers.

-

Post the code, preferably somewhere relatively permanent. (ArXiv, Google code, github, sourceforge, or a permanent departmental page).

Posting the real source code, even if difficult to get running or understand, is the only real hope for someone to reproduce exactly what you have done. Also, it is possible the approach is corrent, but the implementation has errors. There is way to tell this from pseudocode.

Sure, if you have time, an appendix containing pseudocode is nice, but since nobody actually runs pseudocode, its possible that something will be lost in the translation from your sources to the pseudocode.

Of the three, I would think simplified code is the most work for the least benefit. A lot of work, and not much potential gain (are you really going to verify it is correct, and gives the same results as your real code?) All the downsides of pseudocode, without necessarily the clarity.

-
@Kjell, did you mean to say "There is NO way to tell this from pseudocode" in the 2nd paragraph? It would make more sense with a "no" inserted in it... – sleepless in beantown Sep 18 '10 at 21:54

At least for numerical algorithms, submitting it and getting it accepted to ACM Transactions on Mathematical Software ensures that the code is mirrored on both the ACM site and the various mirrors of Netlib.

-