We know that Mobius transformations, $z\to\frac{az+b}{cz+d}$, permutes circles and lines in the Euclidean plane, $(\mathbb{R}^2, dx^2 + dy^2 )$.

It may even be possible to write an explicit formula for the general Mobius action on any given circle:

$$ |z-z_0|=r \mapsto \left|\frac{az+b}{cz+d} -z_0 \right|=r$$

Such a space can be generated by translations $z \mapsto z + z_1$, rotations $z \mapsto \omega z$, dilations $z \mapsto rz$ and inversions $\displaystyle z = \frac{1}{\overline{z}}$. The action on all circles is clear except for the last case:

$$ |z-z_0|=r \mapsto \left| z - \frac{\tfrac{1}{2}z_0}{|z_0|^2 - r^2} \right|= \frac{\tfrac{1}{2}r}{|z_0|^2 - r^2} $$

Is there a more succinct way to write this transformation as a Lie group action?

Here, the Mobius group $G = PSL(2, \mathbb{C}) \simeq SO(1,3)^+$ should act on on the space of circles (and lines) in the plane $\mathbb{R}^2$.

This question also leads me to wonder what the space of circles in the Euclidean plane should be. Naively, the circles are a copy of $\mathbb{C} \times \mathbb{R}^+$ which looks like it could possibly be Hyperbolic space $\mathbb{H}^3$, in which case there would even be a natural metric.

However, the lines all have infinite radius. In which case, we still have three parameters, now a copy of $\mathbb{C} \times S^1$ identifying each line with the closest point to the origin and its direction. The $S^1$ behaving something like the circle at infinity (YouTube).

So now my space of circles is $\big(\mathbb{C}\times \mathbb{R}^+ \big) \cup \big(\mathbb{C}\times S^1 \big)$. How does that happen?

In fact this space should be $\big(\mathbb{C}\times \mathbb{R}^+ \big) \cup \big(S^1 \times \mathbb{R}^+ \big)$.

Introduction to Möbius Differential Geometryby Hertrich-Jeromin andLie Sphere Geometryby Cecil. Personally, I prefer oriented circles and allow Mobius transformations to reverse the orientation, then the answer of Bryant is good enough. $\endgroup$ – Hao Chen Dec 11 '14 at 16:43