Is there a category (in the category theory sense) of nonwellfounded sets (something analogous to Set, the category of sets), and has it been (well)studied? Any references are appreciated.

$\begingroup$ Thank you Neel and Mike for your answers. I found the nLab notes helpful, as well as <a href="plato.stanford.edu/entries/nonwellfoundedsettheory/… section</a> of the SEP article. Sorry, though, I think I meant to ask a more openended question, perhaps unsuitable for mathoverflow. I am interested in references where a category of nonwellfounded sets is used as an alternative to the category of sets. $\endgroup$– Noam ZeilbergerFeb 4, 2010 at 22:41
3 Answers
Yes, there is. See Peter Aczel's 1988 notes on models of nonwellfounded set theory, here. The basic idea is that you can model sets as graphs (ie, as a collection of element sets together with a binary relation intuitively suggesting membership), and then the foundation axiom corresponds exactly to the assertion that the membership relation is wellfounded. Allowing nonwellfounded sets corresponds to allowing arbitrary relations.
These graphs are perfectly ordinary mathematical objects, and so we can collect them into a category exactly the way we do with "normal" sets. The nLab has some notes on this on their page on pure sets, and also on their page on material set theory.

1$\begingroup$ I was aware of Aczel's notes, but the nLab links were helpful, and this answers the title question. My question was meant a bit more openendedly, though, especially whether a category of nonwellfounded sets has been studied/used systematically, like say in categorical logic. $\endgroup$ Feb 4, 2010 at 22:48
Starting from any model of a membershipbased set theory, be it wellfounded or illfounded, you can construct a category of sets and functions. The basic properties of this category (e.g. it is a wellpointed topos) don't depend on whether the model you started from was well or illfounded. In fact, in the presence of the axiom of choice, every illfounded set is still wellorderable, hence bijective to a wellfounded set (a von Neumman ordinal)  thus the category of sets obtained from a model of illfounded set theory + choice is equivalent to the subcategory obtained from its submodel of wellfounded sets.
You can then ask whether you can reconstruct a model of membershipbased set theory from its category of sets; here is where the graphs come in (to model hereditary membership relations), as at nlab:pure set. You can choose to use wellfounded graphs or illfounded ones. If the one you choose matches the type of set theory you started from, then (as long as your set theory is strong enough otherwise) you'll reconstruct the same model. If you choose wellfounded graphs starting from an illfounded set theory, then you'll reconstruct the submodel of wellfounded sets, reproducing the proof of relative consistency of the axiom of foundation. And if you choose illfounded graphs starting from a wellfounded set theory, you'll reproduce Aczel's original proof of the relative consistency of the antifoundation axiom.
According to this doctoral thesis http://www.andrew.cmu.edu/~awodey/students/hughes.pdf by Jesse Hughes (supervised by Steve Awodey), cogebras = coalgebras are the appropriate category to study nonwellfounded sets.