26
$\begingroup$

I would like to know as curiosity how the editorial board or editors* of a mathematical journal evaluate the quality, let's say in colloquial words the importance, of papers or articles.

Question. I would like to know how is evaluate the quality of an article submitted in a journal. Are there criteria to evaluate it? Many thanks.

I think that it must be a difficult task to evaluate the quality of a mathematical paper due how is abstract (the high level of abstraction of research in mathematics) the work of professional mathematicians. Are there criteria to evaluate it, or is it just the experience, knowledges and good work of the people working as publishers?

I'm curious about it but I think that this is an interesting and potentially useful post for other. As soon as I can I should accept an answer.

If there is suitable references in the literature about how they do this work, feel free to refer the literature, answering this question as a reference request, and I try to search and read it from the literature.

*I don't know what is the role of each person working in the edition of a mathematical journal.

Improve this question
$\endgroup$
5
  • $\begingroup$ Thanks in advance to all people adding contributions to this post, I hope that my post is suitable for MathOverflow. $\endgroup$
    – user142929
    Feb 25, 2020 at 10:45
  • 2
    $\begingroup$ I think this works similarly to how it works in other disciplines. "Interesting" roughly means "useful for what active communities of mathematicians are working on", and is thus not a static or timeless thing. This is a necessity of the economy of ideas. It leads to some obvious political problems, but otherwise imagine the difficulty of identifying novelty in a reasonable time... $\endgroup$
    – Jon Bannon
    Feb 25, 2020 at 16:11
  • 2
    $\begingroup$ @JonBannon I agree, it's very subjective and dependent on the surrounding mathematical developments, but I think one can be a slightly more specific: (1) To what extent does it address a problem/methods that others are interested in? (2) What potential does it have to inspire future research? $\endgroup$
    – Kimball
    Feb 26, 2020 at 2:30
  • $\begingroup$ Aside from literally having the word "mathematics" in it, how is this about research-level mathematics? It seems that the same question could be asked almost literatim about pretty much any discipline. $\endgroup$
    – LSpice
    Mar 5, 2020 at 22:05
  • $\begingroup$ I think that this post @LSpice is potentially useful or interesting for many mathematicians, in particular for those who start to research in mathematics. $\endgroup$
    – user142929
    Mar 6, 2020 at 7:36

3 Answers 3

Reset to default
37
$\begingroup$

Assume we are talking about a good journal with a large editorial board representing a wide scope of mathematical interests. I will describe both the role of the editors and the role of the referees. This is my personal viewpoint and others might have different opinion/experience.

The role of the editors. Good journals can accept only about 20% of submitted papers. This is not easy to reject 80% of papers and it often results in rejecting really good papers. Everyone understands that. The procedure of evaluating the papers by the editors is more or less as follows:

The editors have many years of research experience and (hopefully) developed a good mathematical taste. If the paper is close to the research interest of the editor then he or she can relatively easily identify the papers that are not particularly interesting. The reason for not being interesting can be based (for example) on the following criterion:

The result is not well motivated. It is very technical and follows more or less standard arguments. The authors simply take a known result, and prove a new result by slightly modifying the given assumptions. Usually it means that they make the statement more complicated and in a sense more general. Often, they neither have interesting examples supporting such generalizations nor indication of possible applications.

Unfortunately, most of the papers fell into this category. If the editor is sure that this is the case, then he or she rejects the paper without sending it to a referee. Then the authors usually get a rejection notice similar to this one:

We regret that we cannot consider it, in part because at present we have a large backlog of excellent articles awaiting publication. We are thus forced to return articles that might otherwise be considered.

If the editor is not sure about the quality of the paper, then they ask an expert (or several experts) for a quick opinion:

I wonder if you could make a quick, informal assessment of it. Are the results strong enough to warrant sending the article to a referee? Because of our backlog, we like to send to referees only articles that appear to be of very good to outstanding quality.

In that case the expert evaluating the paper is not asked to check all the proofs but to make a judgement based on the criterion explained above. This is an easy task for an expert. If the expert writes a negative opinion, then the authors receive a rejection notice often phrased the way as the rejection notice listed above.

For top journals all experts have to write a positive opinion before the paper is sent to referees.

If the experts' opinion is positive, then the paper is sent to a referee or to many referees. The most extreme case that I know of was a panel of 12 referees who took several years to evaluate the paper (this was when Thomas Hales proved the famous Kepler conjecture). For top quality journals all referees must write positive reports before the paper is accepted (once I received 6 reports, 4 positive and 2 not so positive and the paper was rejected).

Let me also add that the editorial boards are structured basically in two different ways. (1) The authors are asked to choose an editor from the list of editors and submit the paper directly to them. Then the editor who receives the paper handles the submission process according to the rules explained above. (2) The authors submit the paper to the main editor or just to the journal and then the main editor either rejects the paper by themselves or he/she sends it to one of the editors from the editorial board and that editor applies the rules listed above.

Of course some of the journals might have a slightly different approach than the one explained here. There is no a canonical solution and what I wrote is a somewhat a simplified version of the process that is applied in reality.

The role of referees. A paper passed through an initial screening and it was sent to a referee. This is the most unpleasant part of the process. A referee spends a lot of time to read the paper, they are not paid for this job and since their work is anonymous, they do not get any recognition for what they do.

What is the referee required to do? First of all, the referee has to assess originality of the results and whether the results are interesting enough. This part is the same as the one in the initial screening when the paper is sent to an expert for a quick opinion. Secondly, the referee is required to read the paper and check details. Let's be clear about that. Unless the paper is directly related to the research of the referee and he or she really wants to understand the details, there is no way the referee can check all details. Since I cannot speak for other people, let me say what I do in this situation. My answer will only be a simplified version of the real process of the refereeing a paper, just a main idea of what I do.

I go through the whole paper (or most of the paper) to have a good idea of what it is all about, to see a big picture not only of the meaning of the theorems, but also a big picture of the techniques used in the proof. Then I check carefully details of many/some arguments while for other arguments I briefly skim over. If the argument seems reasonable and believable to me I do not bother checking it very carefully. If all details that I check are correct and if all other arguments seem reasonable I am content. In this case, if I like the statement of the main result, I accept the paper. If however, some arguments seem fishy to me, then I check them carefully. This is a point where often I ask the authors for further clarifications. If I really cannot pass through the paper, because I think it has mistakes or if it is written in an unreadable way, I often reject the paper. The biggest problem is when I am convinced that the result proved in a paper is of an outstanding quality, but the paper is very difficult and for that reason not very easy to read. Then, hmm... Then, it is not easy and I often struggle with making a right decision.

Improve this answer
$\endgroup$
2
  • 3
    $\begingroup$ This answer is informative, but I feel like it doesn't clearly express the referees' role in evaluating the quality. $\endgroup$
    – Kimball
    Feb 26, 2020 at 2:24
  • 2
    $\begingroup$ @Kimball Good point. The question was mostly about the role of the editorial board, but you are right I should also say about the role of referees. I will try to do it later when I will have time. $\endgroup$
    – Piotr Hajlasz
    Feb 26, 2020 at 2:28
21
$\begingroup$

If you move $\epsilon \geq 0$ forward in the working area of some influential editors following his/her ideas and citing a lot of his/her papers, that's a fantastic paper! If you stride in an area that is not well-perceived (or well-known) by the editors, that's a minor and uninteresting paper.

Otherwise, you can also solve a one-hundred years open question on which hundreds of mathematicians have worked on without success. Everyone would agree that it makes a great paper. But it seems these days that papers of very high quality are published much more often than interesting questions do appear...

Improve this answer
$\endgroup$
5
  • 3
    $\begingroup$ What do you mean by your last sentence? $\endgroup$
    – user2520938
    Feb 25, 2020 at 11:41
  • 5
    $\begingroup$ @user2520938 : I think Libli is writing in a tongue-in-cheek style. "Papers of very high quality" is a sarcastic phrase, referring to epsilonic improvements in areas of interest to the powers that be, while "interesting questions" are major conjectures on the level of the Riemann hypothesis. $\endgroup$
    – Timothy Chow
    Feb 25, 2020 at 15:52
  • 9
    $\begingroup$ +1 for using $\epsilon \geq 0$ rather than $\epsilon > 0.$ Actually, I think in some (nontrivially many) cases $\epsilon < 0$ applies, especially if one factors in criteria such as @Piotr Hajlasz's observation: "Usually it means that they make the statement more complicated and in a sense more general $[\cdots]$ they neither have interesting examples supporting such generalizations nor indication of possible applications". $\endgroup$
    – Dave L Renfro
    Feb 25, 2020 at 17:37
  • $\begingroup$ @TimothyChow that's what I thought, but I wasn't sure. $\endgroup$
    – user2520938
    Feb 25, 2020 at 17:41
  • $\begingroup$ @DaveLRenfro : Thanks! That's somehow the crux of my "proof" :p $\endgroup$
    – Libli
    Feb 25, 2020 at 18:29
5
$\begingroup$

(i) I think almost every journal has a declared "Aims and scope" policy, and how well the submitted paper fits that policy is perhaps the important criterion in the editors' judgment of how good the paper would be for the readers of the journal. (ii) Obviously, for any reputable mathematical journal, the mathematics in the paper must be correct and nontrivial.

Nowadays, in the era of citation indexes, the main criterion, after the criteria (i) and (ii) above, seems to be how "topical" the paper is, that is, how high is the level of current interest the paper may attract.

Improve this answer
$\endgroup$
2
  • $\begingroup$ So you’re suggesting that it helps for a paper to be sufficiently modish? $\endgroup$
    – Lubin
    Feb 25, 2020 at 15:29
  • 2
    $\begingroup$ @Lubin : I am not suggesting that one should write modish papers. However, if a paper is "topical", it does seem to significantly increase its chances to be accepted, in most journals. $\endgroup$
    – Iosif Pinelis
    Feb 25, 2020 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.