what is the three parameter family of plane projective transformations which fix a unit circle at the origin(that is map the unit circle to itself)? I understand that one such transformation is a rotation but that accounts for just one parameter, what are the other two?

2$\begingroup$ One simple way to see those projective transformation is as isometries of the hyperbolic plane, identified with the unit disk by the Klein model. Since this model is projective, hyperbolic isometries act projectively, leaving the boundary circle invariant. The converse also holds. So, the projective transformations you're looking for are hyperbolic translations, considered in the Klein model of the hyperbolic plane. $\endgroup$ – JeanMarc Schlenker Jul 9 '11 at 17:58
Plane projective transformations can be viewed as linear operators in a 3D vector space V. A circle (more generally an ellipse) in the projective plane P(V) corresponds to a cone C in V. Such a cone is the vanishing locus of an indefinite quadratic form Q on V. Thus the operators we look for are operators preserving Q. In physical lingo, V is a 3D Minkowski space and the operators are Lorentz transformations. They can be described as follows. Choose a basis e1, e2, e3 for V in which Q takes canonical form
1 0 0
0 1 0
0 0 1
One family of Qpreserving operators is the rotations you mentioned:
1 0 0
0 cos(alpha) sin(alpha)
0 sin(alpha) cos(alpha)
Another family consists of hyperbolic rotations (in physical lingo, boosts):
cosh(theta) sinh(theta) 0
sinh(theta) cosh(theta) 0
0 0 1
Analogously, we have hyperbolic rotations in another plane:
cosh(theta) 0 sinh(theta)
0 1 0
sinh(thata) 0 cosh(theta)
We can form a product of 3 matrices, with one of each family. This will yield a general operator of the desired form. This is completely analogous to decomposition of 3D rotations into rotations around separate axes (i.e. Euler angles). Note that 3D rotations are operators preserving a definite quadratic form