56
$\begingroup$

Every time I referee a paper, I dream of a system which would allow me to ask the author a question without troubling the editors. It would save time for everyone involved, most importanly the referee; in my case it would probably half the refereeing time.

Here is the system I envision: together with the paper, the journal would give the referee a link to a secure web form where the referee could type a question which would automatically be emailed to the corresponding author, who in turn could reply in a similar web form. Of course, the editors should have access to the logs of these exchanges, and the system should be set up so that the exchanges (if any) can only be initiated by the referee.

Question. Has this ever been implemented? Are there reasons this cannot (or should not) be implemented in mathematics journals?

Improve this question
$\endgroup$
28
  • 22
    $\begingroup$ I cast a vote against closing. $\endgroup$
    – Andrés E. Caicedo
    Commented Jan 2, 2011 at 19:55
  • 8
    $\begingroup$ When I'm refereeing, I often have the urge to create a fake gmail account and email the author my questions. However, I've never had the guts to actually do this (and probably never will). $\endgroup$
    – Andy Putman
    Commented Jan 2, 2011 at 21:11
  • 7
    $\begingroup$ As for Igor's question (or rather, proposal), of course I sympathize with it, but I reject its very premise - that we need to fix the peer review system of mathematical journals, rather than replacing it by a completely new one. What comes to my mind is, for instance, an arXiv-like database with phpBB2-like discussion, MO-like voting, and some additional structure such as the following one: (1) People with high rep or academic endorsement can have their comments appear directly next to the paper, so instead of the rather small information that we get from a journal publication ("this paper... $\endgroup$
    – darij grinberg
    Commented Jan 2, 2011 at 21:45
  • 20
    $\begingroup$ @Darij: given some of the low-quality stuff I see on the arXiv, and read about as appearing in the proceedings of a faraway country of which I know nothing, do you really want a publishing model where everything is discussed and voted on? and do you believe that the highest-voted questions and answers on MO are really 7 times better than others? $\endgroup$
    – Yemon Choi
    Commented Jan 2, 2011 at 21:47
  • 9
    $\begingroup$ I'm not a big fan of anonymity, but if you want junior faculty at all involved in the refereeing process, it is indispensable. How are you going to get someone's candid opinion otherwise? $\endgroup$
    – Thierry Zell
    Commented Jan 2, 2011 at 21:49

7 Answers 7

Reset to default
15
$\begingroup$

I did exactly as others have suggested: created a fake email account (this was pre-Gmail; I think I used "sithmail") called [author name]reviewer, and corresponded that way.

Actually, I did this not to ask about the paper, but to strongly recommend to the author that he put his article on the arXiv before signing anything (and pointed out that the editor himself frequently posted to the arXiv). I didn't see any point in putting the editor into a difficult position, about whether he was allowed to suggest to the author that the author arXiv the article.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Nice hack! $\endgroup$
    – darij grinberg
    Commented Jan 2, 2011 at 22:36
12
$\begingroup$

For many years now, I have refereed only non-anonymously. It has not been uncommon that the authors of a paper I recommended with reservation—or, especially those I rejected—to write me directly, and we subsequently engage in an online (but off-journal) dialog to improve their paper, often for another venue. Almost uniformly these dialogs have been productively positive. Easy for me to say, because of course I hold all the power strings.

Nevertheless, what Igor B. suggests seems to me would be a significant improvement on the current system. My experience with the more one-sided communication I have experienced predicts that it could work well.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ It would be desirable to have a system that would work regradless of personal bravery. Also I know for a fact that some editors consider an anonymous referee (or anonymous recommendation letter writer) to be much more objective. $\endgroup$
    – Igor Belegradek
    Commented Jan 3, 2011 at 4:37
6
$\begingroup$

The review system for research projects in Italy (during the last few years; it is possibly going to change now) was based on comparing the opinions of two different anonymous referees. At some point the two referees have to exchange opinions with each other and reach an agreement, while remaining anonymous. In practice, this is achieved by channeling communications through a website based at a central location. I post my comments, the other referee can read them and post his comments, etc. Since we both know that our exchange is watched by some official at the Ministry, we feel some pressure as to not revealing our identity. I think this works pretty well, and should work also for peer review.

Actually, several journals have already an electronic system for reports; at the moment it is a little awkward to use it as a communication tool, because of the many steps involved (the messages must be approved through the chain referee->editor->chief editor->editor->author or something like that), but it should be quite easy to make it faster and transform it into a supervised direct channel between referee and author.

Improve this answer
$\endgroup$
5
$\begingroup$

Anyone using one of the suggestions based on "create a fake free email account" should watch out for the possibility that the free email provider will include your IP address in the message. If it does, it is trivial for the recipient (here, the author) to determine which university owns that IP address, which would probably let them make a very good guess at your identity (check the faculty web page for Podunk State University and see who is the expert on reticulated splines). So that would give only the illusion of anonymity.

For example, Yahoo Mail does include the sender's IP address. Gmail apparently does not do so currently, or at least doesn't include it in any obvious way (it could be encoded in some way that is well known but not by me).

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ TOR. $\endgroup$
    – darij grinberg
    Commented Jan 3, 2011 at 22:46
5
$\begingroup$

I think nothing must be changed in the present procedure.

1) We send a paper to a journal when we think that the paper is ready for publication, if it is not, we don't send it. We spend more time on it, if necessary we left the paper 1 or 2 weeks in our drawer before sending it, just in case.

2) Once the paper is in the hands of the referee, the referee judges a work assumed to be complete, not a draft. If it is not accepted the first time but may be improved, it's referee's job to tell what to do and the journal board's job to be the intermediary.

3) After one or two rounds the paper is definitely accepted or rejected. That's the way it works.

The referee is not a teacher, he is a peer and judges as a peer.

Improve this answer
$\endgroup$
5
  • 11
    $\begingroup$ I only said that my job as a referee would be much easier if I could ask questions with no hassle. Currently, I often do not ask questions because this is complicated, and involves troubling busy people. As a result I waste a lot of time. The existing system surely works, but it does so at the expense of people like me. $\endgroup$
    – Igor Belegradek
    Commented Jan 2, 2011 at 22:25
  • 6
    $\begingroup$ Also at the expense of the readers, who would mostly profit from the unasked questions (were they asked). $\endgroup$
    – darij grinberg
    Commented Jan 2, 2011 at 22:37
  • $\begingroup$ I am also a referee, sometimes, and I prefer the way it works, don't take offense :-) If you think that something is wrong in a paper but the paper is good just write down your suggestion (it happened to me recently, I accepted after one round), if you involve too much time, you are a professional, then the paper is not good, reject it. It may be also because you are not the right person to refer the paper, just send it back. That is my way ;) $\endgroup$
    – Patrick I-Z
    Commented Jan 2, 2011 at 23:03
  • 1
    $\begingroup$ The referee-author comminication I suggested would be an extra feature, which a referee may or may not use; those comfortable with the current system could referee exactly as before. $\endgroup$
    – Igor Belegradek
    Commented Jan 3, 2011 at 4:24
  • $\begingroup$ Hi Igor, I understand your position. I had to contact, in the past, a young author to improve the redaction of his paper, the result deserved to be published but the paper was a little bit messy. I found a way to do. My point is that it everyone in this situation may find his way, according to some elementary rules of deontology. It may be not necessary to formalize that in the process of referee, I am afraid that it will complicate a rather simple process, we are used to, without a real gain. But this is just my opinion :) $\endgroup$
    – Patrick I-Z
    Commented Jan 3, 2011 at 16:42
1
$\begingroup$

My opinion is based much more on an idealization of the process of refereeing, rather than the actual process itself, so this opinion may have little worth. It appears to me to have much common sense, so I will share it anyway. (This may all be in whatever procedure or policy manual for referees the journal has.)

The goal of the process is to have someone spend resources putting ink to paper. There are other goals, such as scientific progress, but let us take the economic viewpoint. The representative of the people spending these resources is the editorial team. The team is involved in the process of deciding how to arrange the ink. Your questions and eventual evaluation are key in this decision. I suggest that you send a dated letter (or email) which they can easily extract and forward to the author. In short, they need to be actively involved in the forwarding of such questions.

There are several techniques you can use which ease the process. For one, leave any reference to yourself out of the letter, so the editorial team can put such in if needed.

If you leave whitespace between the questions, you encourage the answerer to fit their answer in that space. Having the answerer meet the challenge of giving a concise answer usually promotes proper answering, and adds value to the process.

Nowadays, many people use a computer to search for a substring. Even so, use every appropriate landmark to help locate the relevant text, as well as an appropriate string, e.g. "Near the end of your second section, just before Lemma C, you have the phrase '...'. What were you thinking when you included that phrase?" .

With the letter to the author, include remarks intended only for the editorial team, saying what will happen if the questions get answered promptly, e.g. your referee report will come six weeks earlier, or you anticipate a particular problem which may evaporate if the right answers come, otherwise you will recommend the problem be fixed. If the editorial team knows explicitly how passing the letter on will benefit them, they will be more motivated to pass it on and save time.

Arrange it so that you only need to send two or three such letters at most. This will involve phrasing your questions properly and accumulating them so that this part of the process is economically advantageous. Also, while becoming a referee, make sure you understand and agree with enough of the editorial process, so that it will allow timely transmission of such questions and answers.

Gerhard "Ask Me About System Design" Paseman, 2011.01.02

Improve this answer
$\endgroup$
2
  • $\begingroup$ "Arrange it so that you only need to send two or three such letters at most." This is not always possible (answers to questions often provoke new questions) and often far from desirable (some papers are considerably improved in proof - this happened with some of mine, for example -, and improving a paper is quite difficult if you can just exchange 3 letters...). $\endgroup$
    – darij grinberg
    Commented Jan 2, 2011 at 22:52
  • $\begingroup$ darij, the real world is different from my ideal, of course. If two or three letters results in significant improvements, then I view it as a stage where the old version is rejected, and the process begins anew with the improved version, and another two or three letters as needed. Of course, the real world will do what it wants anyway, common sense or no. From an economic standpoint, two or three letters seemed appropriate to place in my idealized version. Gerhard "Ask Me About System Design" Paseman, 2011.01.03 $\endgroup$
    – Gerhard Paseman
    Commented Jan 3, 2011 at 9:22
0
$\begingroup$

How about this:

  • First adopt Andy Putnam's suggestion in the comments above: The referee sets up a pseudonymous gmail account to correspond with the author.

  • If the referee's anonymity matters, one would want to use a different pseudonym for each new paper one referees. Otherwise, even if one can't be identified, authors might figure out that the referee named Rollland is the same person as the referee of some other paper, also named Rollland.

  • Some system (a web site) would be available to pick fake names from (and tag them as already taken, if they are) so that the referee's particular personal style of choosing various fake names would not get decrypted. Maybe just 9-digit numbers or the like.

  • If many referees do this and editors object, then their standard boilerplate requests to referee might include their policy that this is not to be done.

All this assumes refereeing continues to be single-blind. The desirability of blind refereeing is debatable. And it won't work with double-blind refereeing unless the editor assists somehow.

Improve this answer
$\endgroup$
12
  • 1
    $\begingroup$ What would force the author to answer any questions from a fake google account? Questions from a referee cannot be ignored. $\endgroup$
    – Igor Belegradek
    Commented Jan 2, 2011 at 22:15
  • 1
    $\begingroup$ @Igor : Of course, you would identify yourself as the referee... $\endgroup$
    – Andy Putman
    Commented Jan 2, 2011 at 22:54
  • 4
    $\begingroup$ Long ago, the Proceedings of the AMS experimented with double-blind refereeing. I refereed one paper that way, but I had, just a few weeks before, heard the author speak on that work. So my report began with something like "If the author is X, accept subject to the following suggestions; if the author isn't X, put him in touch with X, who presented the same results at ..." $\endgroup$
    – Andreas Blass
    Commented Jan 3, 2011 at 2:43
  • 3
    $\begingroup$ @Michael: I have several concerns with the "fake google account" idea. First, the communications are neither approved not supervised by the journal. Second, the author may not believe that this is not a prank, and there is a real referee on the other side. Third, a message from a fake account might never pass the spam fitler. $\endgroup$
    – Igor Belegradek
    Commented Jan 3, 2011 at 4:32
  • 1
    $\begingroup$ Double-blind refereeing seems problematic, because it would be easy to take a good guess at the author by seeing whose name appears most often in the references. And you can't very well blind those, because the referee needs to be able to read the cited papers. How did Proc. AMS deal with this? $\endgroup$
    – Nate Eldredge
    Commented Jan 3, 2011 at 8:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .