29
$\begingroup$

I subscribe to feeds from the arXiv Front for a number of subject areas, using Google Reader. This is great, but there is one problem: when a new preprint is listed in several subject categories, it gets listed in several feeds, which means I have to spend more time reading through the lists of new items, and due to my slightly dysfunctional memory, I often download the same preprint twice. Is there a way to get around this problem, by somehow merging the feeds, using a different arXiv site, or using some other clever trick?

(Hope this is not too off-topic, I think a good answer could be useful to a number of mathematicians. Also, I would like to tag this "arxiv" but am not allowed to add new tags.)

$\endgroup$
  • 6
    $\begingroup$ I think this is a good example of an appropriate meta-mathematical question. $\endgroup$ – David Zureick-Brown Oct 16 '09 at 0:47
24
$\begingroup$

Unless the arXiv has changed recently, articles are published daily which means that the feeds and the email are completely in step.

The problem with the duplicates is that each feed is a separate request to the arXiv for information. The arXiv doesn't know that you are going to merge these results, and I've never heard of a feed reader that attempts to merge feeds to remove duplicates.

However, all is not lost. The feeds that the arXiv provides are not the only way to find information. The arXiv has an API which means that you can effectively craft your own feed. For example, if you point your browser at:

http://export.arxiv.org/api/query?search_query=submittedDate:[20091014200000+TO+20091015200000]&start=0&max_results=500

then you get all the papers submitted yesterday. You can filter your search by subject.

http://export.arxiv.org/api/query?search_query=%28cat:math.AT+OR+cat:math.CT%29+AND+submittedDate:[20091014200000+TO+20091015200000]&start=0&max_results=500

Because the requests are handled all at once, there are no duplicates produced (as can be seen since Emily Riehl's paper is both math.AT and math.CT).

The only catch is that you need to put the date in proper form each time, you can't put in dates such as "today" or "yesterday". Plus the timezone handling is a little weird: the arxiv publishes updates at a certain time determined by the local timezone, which includes daylight saving changes, but the API uses GMT/UTC. So if you want to exactly replicated the "new preprints" announcement of the arxiv then you need to do some funky timezone conversions.

However, this can be done and I've done it. I use a program called RefBase for organising my references and I've modified it so that each morning it presents me with a list of what's new on the arxiv for me to scan through and decide which articles to add to my own bibliographic database. I can also scan back a few days if I've been on holiday. Buried in this extension is the code for figuring out what the date-stamp should be. I could extract it if there's any interest.

Documentation on the arxiv API is at their documentation site. The 'submittedDate' stuff isn't covered there though, that's a newer feature.

| cite | improve this answer | |
$\endgroup$
9
$\begingroup$

Scirate does exactly this. You register there and pick your favorite arXiv sections, and you'll get all papers in those categories with no duplicate everyday. You can browse through past feeds, too. Scirate does not publish Atom or RSS feeds. What they call "feeds" are simply listings on a web page.

I've been using Scirate for math.CO, cs.IT, and puant-ph and never seen duplicates of cross-posted preprints.

You can also "upvote" (or "scite" in their jargon) a preprint and leave a comment. I don't think there are many mathematicians there; it was originally for quantum information folks. But it's already useful if you're alone in your field, and it may prove even more useful if more and more peers in your field start using it, I think.

| cite | improve this answer | |
$\endgroup$
8
$\begingroup$

Admittedly old fashioned, but I get my arxiv fix via email. Over email, if you sign up to several repositories, you get a single combined message, without duplicates.

On the downside, that email comes once a day, so if you're obsessed with the latest, you'll find yourself slightly out of date relative to RSS readers.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Thanks for the quick reply, but I tried the emails before, and I just find the feeds much more convenient, as they are quicker to skim through, and you can star interesting items that you want to come back to later. $\endgroup$ – Andreas Holmstrom Oct 16 '09 at 0:08
  • $\begingroup$ I agree. I use feeds for exactly that reason and am also annoyed by duplicates. Google reader is an easy way for me to organize everything. $\endgroup$ – David Zureick-Brown Oct 16 '09 at 0:48
  • 1
    $\begingroup$ I'm puzzled. I subscribe to two repositories (math.LO and cs.LO) via email, and I get two separate emails each day. How do you persuade them to combine the messages? $\endgroup$ – Emil Jeřábek Jan 4 '13 at 12:22
  • 1
    $\begingroup$ @Emil: I think Scott meant only that the various subfields of math are merged into one e-mail (and similarly for the various subfields of computer science), not that math gets merged with computer science. $\endgroup$ – Andreas Blass Jan 4 '13 at 14:45
  • $\begingroup$ Ah, I see. That makes sense. $\endgroup$ – Emil Jeřábek Jan 8 '13 at 13:36
2
$\begingroup$

I also use Google Reader in (nearly) the way you suggest - but I have simply subscribed to the "all new maths items" feed. It means that I see about 150 new papers in the feed each day but personally I don't find this to be a massive problem (except when I've been away for a week or so). I flick Reader into "list view" and just scroll down the titles, expanding to see the abstract when I think something might be relevant. The majority of authors choose their title well enough to make this work!

It doesn't take so long to scan this list and I know that on a regular basis I see things outside of the subject categories I'd normally focus on. So I do pick up combinatorics, physics and algebraic geometry papers that I'd otherwise not find if I just stuck to math.QA and math.RT. Starring is nice and quick for holding on to things I might come back to later and I maybe download one or two a day too.

I appreciate that this might not be the answer you were looking for but, much as I love tech, sometimes it's quicker and easier to make the organic bit of the process do a bit more work.

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

I wrote a BASH script for automated downloading of selected arXiv RSS feeds.

That content is deduplicated, then parsed into one of two documents:

  • keyword-matched articles of interest;
  • the remaining articles.

The script can be scheduled to run daily via crontab, or manually executed.

By example, today (Jan 10, 2019) among five arXiv RSS feeds my RSS reader provided 690 entries (including duplicated, cross-posted entries) while my script returned 344 de-duplicated (unique) articles.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Which means at least one article appears thrice. $\endgroup$ – Sylvain JULIEN Aug 22 '19 at 18:25
  • $\begingroup$ No: returned results are deduplicated then parsed into one of two results files. $\endgroup$ – Victoria Stuart Aug 22 '19 at 18:34
  • $\begingroup$ I mean before the process of deduplication as $690>2\times 344$. $\endgroup$ – Sylvain JULIEN Aug 22 '19 at 18:36
  • $\begingroup$ You're confusing my comparison of a RSS reader (Firefox Feedbro add-on: 690 hits) vs. my BASH script (344 hits). $\endgroup$ – Victoria Stuart Aug 22 '19 at 18:43
  • $\begingroup$ I just meant that there are at least 3 hits of the RSS reader that give rise to 1 hit of your BASH script. I know nothing about computer science technology, I'm just applying the pigeon-hole principle. My goal was just to emphasize the usefulness of such alternatives to raw hits given by RSS feeds by insisting on their redundancy as one (at least) article appears at least three times in them. $\endgroup$ – Sylvain JULIEN Aug 22 '19 at 19:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.