Send one of the points to infinity by a Mobius tranformation. Your set of points now has the property that the intersection of any two lines passing from pairs of points in the set is also in the set. Such configurations are either all collinear, dense in the plane, or one of these two exceptional cases:
- A point together with a collection of points in a line
- The four vertices of a parallelogram and the intersection of its diagonals.
The first case cannot be circularly stable as soon as $|X|\geq 6$. This is because if we let $P$ be the apex and $A,B,C,D$ be collinear points in that order, then the circumcircles of $PAC$ and $PBD$ intersect in a point not in $X$. We conclude that if $|X|\geq 7$ and $X$ is circularly stable, then either $X$ is collinear or it is dense in the plane.
To classify all circularly stable sets with $|X|=6$ that are not collinear we must be in the second case above. Let the points be $A,B,C,D$ as vertices of the parallelogram and $O$ the intersection of the diagonals. Now, the circumcircle of $ABC$ and the line $BD$ must intersect at $D$ so $ABCD$ is inscribed in a circle, i.e. is a rectangle. Now looking at the circumcircle of $ABO$ and the line $BC$ we must have $BC$ tangent to this circumcircle so $ABCD$ must be a square. It's easy to check that a square, its center and the point at infinity form a circularly stable set.
There are no circularly stable sets with $|X|=5$ that are not collinear. If there were it would come from the first case. We have the apex $A$ and three collinear points $B,C,D$. The circumcircle of $ABD$ and the line $AC$ intersect in a different point, contradiction. On the other hand, as you mention in the OP, any set with $|X|\le 4$ is circularly stable.
The result I used above is proved in "A dense planar point set from iterated line intersections" by D. Ismailescu and R. Radoicic.