Handling arXiv feeds to avoid duplicates

Question

I subscribe to feeds from the arXiv Front for a number of subject areas, using Google Reader. This is great, but there is one problem: when a new preprint is listed in several subject categories, it gets listed in several feeds, which means I have to spend more time reading through the lists of new items, and due to my slightly dysfunctional memory, I often download the same preprint twice. Is there a way to get around this problem, by somehow merging the feeds, using a different arXiv site, or using some other clever trick?

(Hope this is not too off-topic, I think a good answer could be useful to a number of mathematicians. Also, I would like to tag this "arxiv" but am not allowed to add new tags.)

Answer 1

Unless the arXiv has changed recently, articles are published daily which means that the feeds and the email are completely in step.

The problem with the duplicates is that each feed is a separate request to the arXiv for information. The arXiv doesn't know that you are going to merge these results, and I've never heard of a feed reader that attempts to merge feeds to remove duplicates.

However, all is not lost. The feeds that the arXiv provides are not the only way to find information. The arXiv has an API which means that you can effectively craft your own feed. For example, if you point your browser at:

http://export.arxiv.org/api/query?search_query=submittedDate:[20091014200000+TO+20091015200000]&start=0&max_results=500

then you get all the papers submitted yesterday. You can filter your search by subject.

http://export.arxiv.org/api/query?search_query=%28cat:math.AT+OR+cat:math.CT%29+AND+submittedDate:[20091014200000+TO+20091015200000]&start=0&max_results=500

Because the requests are handled all at once, there are no duplicates produced (as can be seen since Emily Riehl's paper is both math.AT and math.CT).

The only catch is that you need to put the date in proper form each time, you can't put in dates such as "today" or "yesterday". Plus the timezone handling is a little weird: the arxiv publishes updates at a certain time determined by the local timezone, which includes daylight saving changes, but the API uses GMT/UTC. So if you want to exactly replicated the "new preprints" announcement of the arxiv then you need to do some funky timezone conversions.

However, this can be done and I've done it. I use a program called RefBase for organising my references and I've modified it so that each morning it presents me with a list of what's new on the arxiv for me to scan through and decide which articles to add to my own bibliographic database. I can also scan back a few days if I've been on holiday. Buried in this extension is the code for figuring out what the date-stamp should be. I could extract it if there's any interest.

Documentation on the arxiv API is at their documentation site. The 'submittedDate' stuff isn't covered there though, that's a newer feature.

Answer 2

Scirate does exactly this. You register there and pick your favorite arXiv sections, and you'll get all papers in those categories with no duplicate everyday. You can browse through past feeds, too. Scirate does not publish Atom or RSS feeds. What they call "feeds" are simply listings on a web page.

I've been using Scirate for math.CO, cs.IT, and puant-ph and never seen duplicates of cross-posted preprints.

You can also "upvote" (or "scite" in their jargon) a preprint and leave a comment. I don't think there are many mathematicians there; it was originally for quantum information folks. But it's already useful if you're alone in your field, and it may prove even more useful if more and more peers in your field start using it, I think.

Answer 3

Admittedly old fashioned, but I get my arxiv fix via email. Over email, if you sign up to several repositories, you get a single combined message, without duplicates.

On the downside, that email comes once a day, so if you're obsessed with the latest, you'll find yourself slightly out of date relative to RSS readers.

Answer 4

I also use Google Reader in (nearly) the way you suggest - but I have simply subscribed to the "all new maths items" feed. It means that I see about 150 new papers in the feed each day but personally I don't find this to be a massive problem (except when I've been away for a week or so). I flick Reader into "list view" and just scroll down the titles, expanding to see the abstract when I think something might be relevant. The majority of authors choose their title well enough to make this work!

It doesn't take so long to scan this list and I know that on a regular basis I see things outside of the subject categories I'd normally focus on. So I do pick up combinatorics, physics and algebraic geometry papers that I'd otherwise not find if I just stuck to math.QA and math.RT. Starring is nice and quick for holding on to things I might come back to later and I maybe download one or two a day too.

I appreciate that this might not be the answer you were looking for but, much as I love tech, sometimes it's quicker and easier to make the organic bit of the process do a bit more work.

Answer 5

I wrote a BASH script for automated downloading of selected arXiv RSS feeds.

That content is deduplicated, then parsed into one of two documents:

• keyword-matched articles of interest;
• the remaining articles.

The script can be scheduled to run daily via crontab, or manually executed.

By example, today (Jan 10, 2019) among five arXiv RSS feeds my RSS reader provided 690 entries (including duplicated, cross-posted entries) while my script returned 344 de-duplicated (unique) articles.

• script | files: https://github.com/victoriastuart/arxiv-rss
• accompanying research blog post: https://persagen.com/2019/06/10/arxiv-rss.html