Consider first the Unrestricted Axiom of Comprehension
($\exists$y)($\forall$x)(x$\in$y $\leftrightarrow$ $\phi$(x))
and the resulting Russell paradox
y$\in$y $\leftrightarrow$ y$\notin$y
One can certainly understand the early set theorists' concern over the existence of a set y such that y$\in$y.
Consider also Cantor's proof found in his letter to Dedekind (found in van Heijenoort's "From Frege to Goedel" (pp113-117) that the system of "all numbers" (all ordinal numbers) $\Omega$ and its successor $\Omega^{'}$ are "inconsistent multiplicities". In order for his 'proof' to work one must allow $\Omega$$\in$$\Omega$ and the infinite descending sequence ....$\in$$\Omega^{''}$$\in$$\Omega^{'}$$\in$$\Omega$ (obviously for the Burali-Forti paradox as well).
In Zermelo's paper "Investigations in the foundations of set theory I" (also found in van Heijenoort--pp. 199-215), one finds the Axiom of Foundation cropping up as the following "theorem":
"Every set $M$ possesses at least one subset $M_0$ that is not an element of $M$."
which he 'proves' using the Axiom of Separation.
A good introduction to the Axiom of Regularity and its philosophical and historical underpinnings is the Wikipedia entry "Axiom of Regularity".
That having been said, a good (at least in my opinion) introduction to nonwellfounded set theory is the paper by Takashi Nitta, Tomoko Okada, and Athanassios Tzouvaras titled "Classification of non-well-founded sets and an application" which can be found under title on the Web.