Is there a category (in the category theory sense) of nonwellfounded sets (something analogous to Set, the category of sets), and has it been (well)studied? Any references are appreciated.

Yes, there is. See Peter Aczel's 1988 notes on models of nonwellfounded set theory, here. The basic idea is that you can model sets as graphs (ie, as a collection of element sets together with a binary relation intuitively suggesting membership), and then the foundation axiom corresponds exactly to the assertion that the membership relation is wellfounded. Allowing nonwellfounded sets corresponds to allowing arbitrary relations. These graphs are perfectly ordinary mathematical objects, and so we can collect them into a category exactly the way we do with "normal" sets. The nLab has some notes on this on their page on pure sets, and also on their page on material set theory. 


Starting from any model of a membershipbased set theory, be it wellfounded or illfounded, you can construct a category of sets and functions. The basic properties of this category (e.g. it is a wellpointed topos) don't depend on whether the model you started from was well or illfounded. In fact, in the presence of the axiom of choice, every illfounded set is still wellorderable, hence bijective to a wellfounded set (a von Neumman ordinal)  thus the category of sets obtained from a model of illfounded set theory + choice is equivalent to the subcategory obtained from its submodel of wellfounded sets. You can then ask whether you can reconstruct a model of membershipbased set theory from its category of sets; here is where the graphs come in (to model hereditary membership relations), as at nlab:pure set. You can choose to use wellfounded graphs or illfounded ones. If the one you choose matches the type of set theory you started from, then (as long as your set theory is strong enough otherwise) you'll reconstruct the same model. If you choose wellfounded graphs starting from an illfounded set theory, then you'll reconstruct the submodel of wellfounded sets, reproducing the proof of relative consistency of the axiom of foundation. And if you choose illfounded graphs starting from a wellfounded set theory, you'll reproduce Aczel's original proof of the relative consistency of the antifoundation axiom. 


According to this doctoral thesis http://www.andrew.cmu.edu/~awodey/students/hughes.pdf by Jesse Hughes (supervised by Steve Awodey), cogebras = coalgebras are the appropriate category to study nonwellfounded sets. 

