Question

What is the largest subgroup of the diffeomorphism group of a (2n-1)-sphere that only contains members that are holomorphic in the coordinates assigned to the sphere by taking it to be the unit sphere of a complex n-space? Sorry if I'm not being very clear; I will clarify as necessary.

Answer 1

I think that you are asking for the group of automorphisms of the CR-structure on $\mathbb{S}^{2n-1}$ as hypersurface in $\mathbb{C}^n$ (assuming $n>1$, since $n=1$ is another story). Even locally, any such automorphism is induced by an element of $PU(n,1)$, which is the subgroup of $PGL(n+1,\mathbb{C})=\mathrm{Aut}(\mathbb{C}P^n)$ preserving the unit ball $\mathbb{B}\subset\mathbb{C}^n\subset\mathbb{C}P^n$. This result is due to S.S. Chern and J. Moser, Acta Math. 133 (1974), 219--271 (but the global version you are asking for is perhaps older).

Concretely, $PU(n,1)$ is the isometry group of the hermitian form $|z_1|^2+\dots+|z_n|^2-|z_{n+1}|^2$, quotiented by its scalar subgroup $U(1)$. The diffeomorphism of $\mathbb{B}\subset\mathbb{C}^n$ associated to a matrix $A\in U(n,1)$ is given by $v\mapsto (a_{11}v+a_{12})/(a_{21}v+a_{22})$, where $a_{ij}$ are the blocks of $A$ for the obvious decomposition $\mathbb{C}^{n+1}=\mathbb{C}^n\oplus\mathbb{C}$ (thus $a_{22}$ is scalar, for instance, and $a_{21}$ is an $1\times n$ line matrix).

Hope this answers your question

Answer 2

A slightly more elementary question, whose answer may be easier to grasp, is to determine the largest subgroup of the isometry group $O(2n)$ of $S^{2n-1}$ that acts holomorphically on $\mathbb{C}^n$. The answer is the unitary group $U(n)$, that is of course also a subgroup of the automorphism group $PU(n,1)$. The unitary group (and also $SU(n)$) is is one of a fairly restricted list of compact Lie groups that act effectively and transitively on a sphere.