Peer review 2.0

Question

I have an idea for a website that could improve some well-known difficulties around peer review system and "hidden knowledge" in mathematics. It seems like a low hanging fruit that many people must've thought about before. My question is two-fold:

Has someone already tried this? If not, who in the mathematical community might be interested in creating and maintaining such a project or is working on similar projects?

Idea

A website dedicated to anonymous discussions of mathematical papers by experts.

Motivation 1: Hidden knowledge

Wilhelm Klingenberg's "Lectures on closed geodesics" can be found in every university's math library. One of the main theorems in the book is the following remarkable result, a culmination of decades of work on Morse theory of the loop space by many mathematicians: Every compact Riemannian manifold contains infinitely many prime closed geodesics.

Unfortunately, there is a mistake in the proof. 44 years after the book's publication the statement is still a widely open problem. The reason I know this is because when I was in grad school I mentioned the book to my adviser and my adviser told me about it. If I tried to look for this information online I wouldn't find it (in fact, I still haven't seen it written down anywhere).

This is one of many examples of "hidden knowledge", information that gets passed from adviser to student, but is inaccessible to an outsider. In principle, a new Ramanujan can open arxiv.org and get access to the cutting edge mathematical research. In reality, the hidden knowledge keeps many mathematical fields impenetrable to anyone who is not personally acquainted with one of a handful of experts.

Of course, there is much more to hidden knowledge than "this paper form 40 years ago actually contains a gap". But I feel that the experts' "oral tradition" on papers in the field is at the core of it. Making it common knowledge will be of great benefit to students, mathematicians from smaller universities, those outside of North America and Europe, people from adjacent fields, to the experts themselves and to the mathematical progress.

Motivation 2: Improving peer review

Consider the following situations:

• You are refereeing a paper and get stuck on some minor issue. It will take the author 5 minutes to explain, but a few hours for you to figure it out on your own. But it doesn't quite feel worth initiating formal communication with the author through the editor over this and you don't want to break the veil of anonymity by contacting the author directly.
• You are being asked to referee a paper, but don't have time to referee the whole paper. On the other hand, there is a part of it that is really interesting to you. Telling the editor "yes, but I will only referee Lemma 5.3" seems awkward.
• You are refereeing a paper that is at the intersection of your field and a different field. You would like to discuss it with an expert in the other field to make sure you are not missing anything, but don't know a colleague in that area or feel hesitant revealing that you are a referee for this paper.

These are some of many situations where ability to anonymously discuss a paper with the author and other experts in a forum-like space would be helpful in the refereeing process. But also outside of peer review mathematicians constantly find small errors, fillable gaps and ways to make an old paper more understandable that they would be happy to share with the others. At the same time, they often don't have time to produce carefully polished notes that they would feel comfortable posting on arxiv, or if they post notes on their website they may not be easy to find for anyone else reading the paper. It would be helpful to have one place where such information is collected.

How will it work?

The hope is to continue the glorious tradition of collaborative anonymous mathematics. One implementation can work like this: Users of the website can create a page dedicated to a paper and post questions and comments about the paper on that page. To register on the website one needs to fill in a form asking for an email and two links to math arxiv papers that have the same email in them (this way registration does not require verification by moderators) and choose their fields of expertise. When a user makes a comment or question only their field/fields are displayed.

Answer 1

I'm the founder of https://papers-gamma.link, an Internet place to discuss scientific articles, mentioned by Matthieu Latapy. I have been supporting this site for 6 years now. I hope that one day it will become popular (in a good sense of the word) and useful for the entire scientific community. As you may imagine, I'm pretty convinced that the idea of public review and public comments is, potentially, a very promising one. My persuasion is less than 6 years ago though, and here's why.

Observing that Papers$^\gamma$ gains its popularity very slowly, I started to think more and more about the scientific review and publishing processes. I'll share with you my current understanding of this subjects, admitting that these issues are more likely to be in the field of sociology rather than mathematics.

The original goal of scientific journals was to inform about and discuss the research currents. But after, this had to make room for other things: archiving and bibliometrics. I'm OK with archiving. But, it seems that the optimization of bibliometric statistics negatively affects the discussion power, the original (!) goal of the scientific journals.

Let me try to illustrate exactly what I mean by "discussion power" of scientific journals. Consider Miller's one-page paper [2], which is an excellent example of conversational mathematics. The paper contains an alternative proof of Miles' results [1] about the characteristic equation of the $k$-th order Fibonacci sequence. Miller's paper can be considered as a response to Miles's paper. Compare this to the modern Internet forums like MathOverflow.

Does any journal, wiki, or another internet portal is attracted to archiving and bibliometrics as time goes forward? Maybe there is a natural or induced drift from "discussion" towards "judgment"? If it is the case, should we consider to create a new journals every, say, 10 or 20 years, to restart discussion processes? Or should we be extremely careful, trying to keep the discussion power of existing scientific journals?

Here are main arguments against the mass acceptance of public review and comments, that I can imagine:

1. Existing systems works pretty well.

2. Not all conversations should be made public. Private reviews and conversations have their advantages. People feel more free to commit errors, express misunderstanding, and criticize in private. Good reviews help the authors to polish and publish almost perfect articles.

3. The more a resource is open, the more it is susceptible to spam.

4. Anonymous public discussions may be used as a platform for attacking other scientists.

Here is a list of some online resources related to the idea of public reviews and collaborative science in general:

• MathSciNet and Zentralblatt MATH, databases with post-publication reviews.

• nLab wiki and The $n$-Category Café, collaborative works on Mathematics, Physics and Philosophy.

• Polymath Project, a collaborative project that aims to solve important and difficult mathematical problems.

• Machine Learning Paper Discussions subreddit.

• Atmospheric Chemistry and Physics journal with interactive public peer review process.

• CoScience, a service that aims to "recreate scientific communication as a virtuous, open, community-driven process".

• F1000Research, a platform covering the life science publications.

• PubPeer, a online platform for post-publication peer review. "The site has served as a whistleblowing platform, in that it highlighted shortcomings in several high-profile papers, in some cases leading to retractions and to accusations of scientific fraud", as Wikipedia says.

• BibSonomy, a social bookmark service allowing comments.

• Selected Papers Network was an open-source project for share and comment scientific articles.

I wonder if certain usenet groups in sci.* hierarchy are still active.

In any case, the source code of Papers$^\gamma$ is open under CC0 Public Domain Dedication. And you are welcome to send me a patch or fork it if you wish so.

To conclude, I think that "Hidden knowledge" will always be here, and the solution to this issue lies not in the technical but rather in the societal dimension. Someone just need to write it down in some searchable place. For instance, in enumerative combinatorics Sloane's The On-Line Encyclopedia of Integer Sequences helps a lot, but we need to watch out and constantly update it.

Conclusion update: for me, both OEIS and MathOverflow are popular because their main purpose is to allow people make a research together and not to judge each other.

Paywalled Biblio

[1] Miles, E. “Generalized Fibonacci numbers and associated matrices”. The American Mathematical Monthly, 67(8), 1960, 745–752

[2] Miller, M. D. "On generalized Fibonacci numbers". The American Mathematical Monthly, 78(10), 1971, 1108–1109.

Answer 2

• SelectedPapers was created "as a space where academics could read, share, and give feedback on articles and papers related to their field.". It is no longer in operation.
• Scirate is an active discussion forum for papers on arXiv, mostly in the area of quantum information processing. It copies the arXiv subcategories in mathematics, so this could well function in that area as well.

Most contributors on these sites disclose their names, but that is not required. Some journals have implemented an open discussion functionality.

• SciPost is one of these, it welcomes anonymous discussions --- both comments by experts invited as reviewers of a manuscript and also unsollicited comments.
  _{The physics branch of SciPost is well developed, the mathematics branch has just started with two editors, it now needs more editors who are willing to help set up this branch. (Disclosure, I'm a SciPost Physics editor, feel free to get in touch if this is something you might consider doing.)}
• Perhaps most relevant for MathOverflow, I mention that PhysicsOverflow has implemented a review section, where authors can ask the community to comment on their work. The discussion could be a critical appraisal or just a summary of the paper.
  I do note that, unlike MO, PhysicsOverflow is not part of the StackExchange ecosystem, I don't think such functionality could be implemented as part of SE. I also note that the PhysicsOverflow review functionality has not been a success: Less than 10 reviews per year are entered.

Answer 3

Imho, mathematics has already a good culture and a relatively rich informational ecosystem to detect, document, and disseminate these issues (many options are mentioned above); what might be lacking is a kind of (semi)automated system of interlinked communication that helps spreading the information further.

Judging from the several examples we had to deal with at zbMATH Open (and the discussions involved), I would certainly agree that it seems less feasible to have a single seemingly authoritative source, but we need to employ and coordinate our existing procedures. A natural problem would be, e.g., the scope - there is a number of problematic (mostly, APC) journals around where one has doubts about the proper peer review - would you like to have a website that addresses all of them (including simple Fermat/Goldbach proofs etc.)? Likely, serious mathematicians lack the resources to do that.

But also with restricted scope, every single issue is a tedious work. To give a recent example, even in a rather clear-cut case

Ramm, Alexander G., Solution of the Navier-Stokes problem, Appl. Math. Lett. 87, 160-164 (2019). ZBL1410.35100.

it was impossible to convince the author of the reviewer's arguments (although we employed another expert as arbiter who went through the arguments and clearly ruled in favour of the reviewer). Every such issue is different and needs to be handled carefully.

On the technical side, e.g., if we detect such an issue with a widely accepted older article via zbMATH Open (documented usually as an additional "Editorial remark" in the original) which is lacking an erratum/corrigendum for this, I currently post this also manually as an answer at the standard question at MathOverflow

Widely accepted mathematical results that were later shown to be wrong?

which has the additional advantage that an automated backlink in the respective zbMATH entry is generated (e.g., see the entry

Saito, Masa-Hiko, On the infinitesimal Torelli problem of elliptic surfaces, J. Math. Kyoto Univ. 23, 441-460 (1983). ZBL0532.14019.)

Of course, this doesn't mean everyone is aware of this information, and there are likely various ways to improve/automatize things further. One could think about creating a specific MathOverflow entry/tagging for each issue/publication that needs discussion/updates which could trigger an activity on our and other sites. But the main effort will likely always be mathematically, not technically, and require mathematicians who are willing to explore things deeply. The platforms are there, we should just make sure that the documentation is appropriate, accessible, and ideally increasingly interoperable.

Answer 4

It seems to me that Papers$^\gamma$ is close to what you describe: people can upload a paper, inform its authors, and publicly (anonymously or not) discuss it.

I would also like to mention PCI (Peer Community In), "a free recommendation process of scientific preprints based on peer reviews" that published the discussion between authors, reviewers, and editors (or their equivalent); there is a nice video presentation.

Answer 5

I'm one of the founders of info.inquire.pub (main site not yet available to the public). We're a non-profit startup with a similar purpose. We aim to be a Q&A site with a reputation system and upvotes, where each question is associated with a specific paper (we support ArXiv or DOI so far).

We are still deciding exactly how anonymous we want to be. We continually discuss potential issues like those @TerryTao brings up in a comment above and are committed to designing a system which will be robust against trolling and weaponisation.

In my opinion, academic research is one of societies greatest resources. We want to improve the state of research. We want to save researchers time and effort. We'd also like to provided a venue for cross-disciplinary review and for the general public to ask high quality objective questions about academic literature. We take possible negative impacts seriously.

The current peer review process is far from perfect and once articles are published they're rarely every edited again. If anyone would like to help us (including simply joining the conversation), my contact info is available at mathandy.ai.

Answer 6

I'm 100% in agreement about the need for such a site. In fact I'm currently in the process of coding what I hope can grow into such a site.

The apps mentioned above like Papers$^\gamma$ are all nice but they all just kinda throw the user at a list of papers and say: go comment/review. That can work to collect some objections or point out issues but only if people are visiting the site enough to think that's where the comment goes and it's not that helpful to aggregate knowledge about what's a great paper.

The challenge here is that you need a website which is first and foremost useful and engaging enough for mathematicians to visit it on a regular basis. Something like the question system here (or at least regular professional gossip etc) is one obvious possibility but stack exchange already exists so another possibility is a good system for managing collaboration (so like a scratch pad where you can annotate notes someone else made, version controlled uploads of papers etc)

Furthermore, a good website for this purpose should also have

1. The ability to rate or at least upvote articles without saying anything. Comments are nice enough for issues but do little to help other people find the best articles.

2. Some way to feature particular articles that are either hand selected or are well rated by users.

   I'm thinking allowing users with enough karma/trust create a list of articles that they can add to regularly thus creating something like a mini-journal as well as allowing the creation of lists of good references for various subjects.

3. The ability of authors to claim papers and respond to questions about them (both publicly and privately) as well as offer notes to be displayed alongside the paper/link (eg in Lemma 5 n must be at least k+1 not k as written)

I think it would also be desierable to have some way to verify participants as mathematicians (defined broadly to basically mean anyone who can understand a math paper..but it needs to exclude ppl looking for homework help and cranks) which could be easily done via the system arxiv uses of vouching someone else in or even do it on trust initially.

Anyway in some sense this isn't an answer yet but I think it's important to convey that the existing solutions either lack the features that would aggregate information (thinking academia.edu and researchgate) or lack the features which would enable a community to grow up around the site or both.

I'm going to give writing such a site myself a go because I've wanted it to exist for 15 years and so far all the attempts fail at the generating interest level. But I'm likely to fail down to just a fancy blog so I'm mentioning these aspects in the hope someone else will create a site with them.