Let $Q$ be a nondegenerate indefinite quadratic form on ${\mathbb R}^n$ and write $G=SO(Q)$ for the associated special orthogonal group. Let $K$ be a maximal compact subgroup of $G$ and consider the standard action of $H$ on $V$. It is wellknow (and easy to see) that for two vectors $v_1, v_2 \in V$ to be on the same $K$orbit it is necessary and sufficient that $Q(v_1)=Q(v_2)$ and $\ v_1\= \ v_2\$, where $\ . \$ is the norm associated to a suitablychosen inner product on ${\mathbb R}^n$. My question is there is a similar characterization of the orbits, when $Q$ is a nondegenerate quadratic form on the field of $p$adic numbers ${\mathbb Q}_p^n$. More precisely, suppose $Q$ is a nondegenerate isotropic form on ${\mathbb Q}_p^n$, $G=SO(V)$ defined as before, and $K$ is a maximal compact subgroup of $G$. Can one give a neat characterization of the action of the orbits of $K$ on ${\mathbb Q}_p^n$? As far as I know, in the $p$adic setup there many be several conjugacy classes of maximal compact subgroups. In case, the answer to the first question is positive for some maximal compact subgroups, is it possible to realize such a maximal compact subgroup as the intersection of $G$ with a suitable conjugate of $SL_n( {\mathbb Z}_p)$ in $SL_n( {\mathbb Q}_p)$?
Let's first review the situation over $\mathbf{R}$, as it gives an excuse to recall Witt's extension theorem that we will use in the nonarchimedean case and to see that we had better restrict to dimension at least 3. (The assertion made over $\mathbf{R}$ is false in dimension 2: the locus of nonzero vectors where $Q=0$ is a pair of independent lines with 0 removed, and this meets any positive level set of a positivedefinite norm in 4 points, yet the maximal compact of ${\rm{SO}}(Q)(\mathbf{R}) = \mathbf{R}^{\times}$ has only 2 points.)
Write an indefinite nondegenerate quadratic space $(V,Q)$ over $\mathbf{R}$ as an orthogonal sum $(V_{+}, Q_{+}) \perp (V_{},Q_{})$ of (nonzero) positivedefinite and negativedefinite spaces, so $K' := {\rm{O}}(Q_{+})(\mathbf{R}) \times {\rm{O}}(Q_{})(\mathbf{R})$ is thereby a compact subgroup of ${\rm{O}}(Q)(\mathbf{R})$ which is moreover maximal as such, and its determinant1 intersection $K$ with $G = {\rm{SO}}(Q)(\mathbf{R})$ is maximal compact in $G$. But the values $Q_{+}(v)$ and $Q_{}(v)$ are determined by the combined knowledge of $Q(v) = Q_{+}(v) + Q_{}(v)$ and $Q'(v) = Q_{+}(v)  Q_{}(v)$ with $Q' = Q_{+}  Q_{}$ a positivedefinite form, and $K = {\rm{O}}(Q)(\mathbf{R}) \cap {\rm{O}}(Q')(\mathbf{R})$ inside ${\rm{GL}}(V)$.
Witt's extension theorem gives transitivity of the action of the rational points of the orthogonal group of any finitedimensional quadratic space over any field whatsoever on the set of nonzero vectors outside the defect space with a given value (possibly 0) for the quadratic form (see Theorem 8.3 in "The algebraic and geometric theory of quadratic forms" by Elman, Karpenko, Merkurjev). This gives the result for full orthogonal groups when applied to the indefinite $Q$ and the positivedefinite $Q'$ (with special argument to handle cases when a vector has vanishing component along $V_+$ or $V_{}$). It remains to deal with the gap between ${\rm{O}}(Q)(\mathbf{R})$ and ${\rm{SO}}(Q)(\mathbf{R})$; it is at this final step where we have to use that $\dim V > 2$.
More specifically, it suffices to show that the ${\rm{O}}(Q)(\mathbf{R})$stabilizer of a nonzero (but possibly isotropic!) vector $v$ contains a reflection, as then ${\rm{SO}(Q)}(\mathbf{R})$ maps onto the quotient of ${\rm{O}}(Q)(\mathbf{R})$ modulo the right action of the $v$stabilizer (whence the ${\rm{O}}(Q)(\mathbf{R})$orbit of $v$ is the same as the ${\rm{SO}}(Q)(\mathbf{R})$orbit). Note that this is false for isotropic $v$ in a hyperbolic plane over any field.
If $v$ is nonisotropic then $v^{\perp}$ is a (nonzero) hyperplane complementary to $\mathbf{R}v$ on which $Q$ has nondegenerate restriction, so this hyperplane contains a nonisotropic vector and any reflection in such a vector does the job. If $v$ is isotropic then it lies in a hyperbolic plane $H$ in $V$, so $Q$ has nondegenerate restriction to $H^{\perp}$, and $H^{\perp} \ne 0$ (as $\dim V \ge 3$), so $H^{\perp}$ contains a nonisotropic vector; any reflection in such a vector does the job.
We're done over $\mathbf{R}$.
In the nonarchimedean case the answer appears to be negative when the residue characteristic $p$ is even. Consider a split quadratic space $(V,F)$ with odd dimension $d=2n+1 \ge 5$ over a 2adic field $F$ with finite residue field $k$, so $G = {\rm{SO}}_d$ and $G(F)$ has a unique conjugacy class of hyperspecial maximal compact subgroups, namely the class of $K = {\rm{SO}}_d(O_F)$ (since $G$ is split semisimple of adjoint type). The "standard" norm can be viewed as the one defined by the lattice $L = O_F^d \subset F^d$, and the action of $K$ on the reduction $L_0 = k^d$ is through the quotient ${\rm{SO}}_d(k)$.
The group ${\rm{O}}_d(k)$ acts transitively on all unit vectors in $k^d$ outside the defect line, by Witt's theorem (cf. reference above), and since $d \ge 5$ we can find a reflection in the stabilizer of any such vector, so ${\rm{SO}}_d(k)$ also acts transitively. Such unit vectors span $k^d$ as well since $d \ge 5$ (e.g., for $x_0^2 + x_1 x_2 + x_3 x_4$ on $k^5$, some unit vectors are $e_0 + e_1$, $e_0 + e_3$, $e_0 + e_1 + e_3$, $e_0 + e_2$, and $e_0 + e_4$). Consequently, any $K$invariant norm on $F^d$ has unit ball that is a scaling of the standard lattice, yet the $K$action on the set of unit vectors in the unit ball is not transitive because the action of ${\rm{SO}}_d(k)$ on $k^d$ leaves the defect line invariant (so the unique residual unit vector in that line is not carried to any other residual unit vector in $k^d$).
Now turn to a nondegenerate quadratic space $(V,q)$ of dimension at least 3 over a nonarchimedean local field $F$ with residue field $k$ of characteristic $p > 2$. Let $\mathbf{G}$ be the smooth affine $F$group ${\rm{O}}(q)$ with $G := \mathbf{G}^0 = {\rm{SO}}(q)$ connected semisimple. Again by Witt, $\mathbf{G}(F)$ acts transitively on $\{v \in V  \{0\}\,\,Q(v)=c\}$ for $c \in F$ (allowing $c=0$). Suppose there is an $O_F$lattice $L$ in $V$ on which $Q$ is $O_F$valued with $(L,Q)$ nondegenerate over $O_F$ (i.e., its reduction is nondegenerate over the residue field). In such cases we have an affirmative answer to your question. (Maybe someone with more expertise in BruhatTits theory can put this into a wider context.)
Note that the $O_F$group scheme $\mathscr{G} := {\rm{SO}}(L,q)$ is smooth with connected reductive fibers, so its group $K$ of $O_F$points is a (hyperspecial) maximal compact subgroup of $G(F)$, and $K = G(F) \cap {\rm{SL}}(L)$. I claim that the $F$valued norm $\cdot$ on $V$ with unit ball $L$ does the job for $K$orbits on $V\{0\}$.
OK, now consider $v, w \in V\{0\}$ with $q(v) = q(w)$ and $v = w$ (necessary conditions for nonzero $v$ and $w$ to be in the same $K$orbit). We want to show that such $v$ and $w$ are in the same $K$orbit. It is harmless to scale by an element of $F^{\times}$, we so may and do arrange that $v = w = 1$. Hence, $v, w \in L$ with nonzero reduction in $L_0 = L/\mathfrak{m}L$ ($\mathfrak{m}$ the maximal ideal of $O_F$). Note also that now the value $q(v) = q(w)$ lies in $O_F$ (though possibly in $\mathfrak{m}$) since we assumed $q$ is $O_F$valued on $L$. The reductions $v_0, w_0 \in L_0$ are nonzero vectors in the quadratic space $(L_0, q_0)$ that is nondegenerate by hypothesis. By design $q_0(v_0) = q_0(w_0)$ in the residue field $k$ of $F$ (just the reduction of $q(v) = q(w) \in O_F$), though this common value could well be zero (but $v_0, w_0 \ne 0$!).
Now I claim that the transporter closed subscheme ${\rm{Transp}}_{\mathscr{G}}(v,w)$ inside $\mathscr{G}$ is smooth over $O_F$, so if its special fiber has a rational point then we could then lift that to an $O_F$point which does the job. Below we set up and prove this smoothness assertion in a more general setting, as it has nothing to do with discrete valuation rings. Let's now show that there is a residual rational point (i.e., ${\rm{SO}}(q_0)(k)$ carries $v_0$ to $w_0$), so then we'd be done modulo the $O_F$smoothness of the transporter scheme.
Since ${\rm{char}}(k)\ne 2$, the nondegeneracy of $q_0$ implies that the associated symmetric bilinear form $B_{q_0}$ is nondegenerate. Hence, we can again use Witt's theorem (cf. reference above) to deduce that ${\rm{O}}(q_0)(k)$ carries $v_0$ to $w_0$. I claim that ${\rm{SO}}(q_0)(k)$ does the job here too. This just requires showing that the $v_0$stabilizer in ${\rm{O}}(q_0)(k)$ contains a reflection. If $v_0$ is not isotropic then (since $p > 2$) $q_0$ is nondegenerate on the orthogonal hyperplane $(kv_0)^{\perp}$ complementary to $kv_0$, so this hyperplane has nonisotropic vectors, and a reflection in any of those does the job. If $v_0$ is isotropic then $v_0$ lies in a hyperbolic plane $H_0$ inside $V_0$, so the complementary $H_0^{\perp}$ has nondegenerate restriction for $q_0$ and is nonzero (as $\dim V_0 = \dim V \ge 3$), whence $H_0^{\perp}$ has nonisotropic vectors for $q_0$, any reflection in which does the job.
Finally, we address the smoothness of a transporter scheme in general setting. Let $(L,q)$ be a fiberwise nondegenerate quadratic space over a $\mathbf{Z}[1/2]$algebra $A$, and consider $v, w \in L$ that are fiberwise nonzero and for which $q(v) = q(w)$. (We make no assumptions on this common $q$value; it might be 0 or anything else.) Let $G$ be the closed subgroup scheme ${\rm{SO}}(q) \subset {\rm{GL}}(L)$, so $G$ is $A$smooth, and let $X \subset G$ be the finitely presented closed subscheme representing the functor of points of $G$ that carry $v$ to $w$. We claim that $X$ is $A$smooth.
To prove this smoothness we may (by standard direct limit arguments) assume $A$ is noetherian, and it suffice to check the infinitesimal criterion with artin local $A$algebras having algebraically closed residue field $k$ (so ${\rm{char}}(k) \ne 2$). We may therefore assume $A$ is such a ring, say with maximal ideal $\mathfrak{m}$ satisfying $\mathfrak{m}^{m+1} = 0$ with $m \ge 1$, and that we're given an ideal $I \subset \mathfrak{m}$ of length 1 and a point $g_0 \in X(A/I)$ which we have to lift to $X(A)$. Since $I$ has length 1, it is spanned by a nonzero $\mathfrak{m}$torsion element $t \in A$. In particular, $IL = tL \simeq L_0$ as $A$modules.
By smoothness of $G$ we can choose $\widetilde{g} \in G(A)$ lifting $g_0$ and hit $v$ with that element to reduce to the case that $v \equiv w \bmod I L$ (so in particular the nonzero reductions $v_0, w_0 \in L_0 = L/\mathfrak{m}L$ coincide) and $g_0 = 1$. We therefore seek $g \in \ker(G(A)\rightarrow G(A/I))$ carrying $v$ to $w$. Viewing such a hypothetical $g$ inside $\ker({\rm{GL}}(L) \rightarrow {\rm{GL}}(L/IL))$, we may write it (if it is to exist at all) inside ${\rm{End}}(L/IL)$ as $g = 1 + T$ for $T \in {\rm{Hom}}_A(L,IL) = t {\rm{End}}_k(L_0)$. Writing $T = tT_0$ for $T_0 \in {\rm{End}}_k(L_0)$, the condition $g \in G(A)$ (i.e., $g$ preserves $q$) is exactly the condition $T_0 \in {\mathfrak{o}}(q_0) = \mathfrak{so}(q_0)$; i.e., $B_{q_0}(\cdot, T_0(\cdot))$ is alternating. Since $v \equiv w \bmod IL$, we can write $v = w + tx_0$ with $x_0 \in L_0$. The hypothesis $q(v) = q(w)$ says exactly that $B_{q_0}(w_0,x_0) = 0$ where we recall that $w_0 = v_0 \ne 0$ in $L_0$. Finally, the condition $g(v)=w$ says exactly that $T_0(v_0)=x_0$.
Now we are reduced to a problem for the nondegenerate quadratic space $(L_0,q_0)$ and the nonzero (but possibly isotropic) vector $v_0 \in L_0$: the map ${\mathfrak{so}}(q_0) \rightarrow v_0^{\perp}$ defined by $T_0 \mapsto T_0(v_0)$ is surjective. It suffices then to show that the kernel has codimension $d1$, which is to say has dimension $(d1)(d2)/2$. If $T_0(v_0)=0$ then the entire image of $T_0$ is inside $v_0^{\perp}$, so $T_0 \in {\rm{Hom}}(L_0/kv_0, v_0^{\perp})$. (Note that $v_0$ may lie in $v_0^{\perp}$; i.e., $v_0$ might be isotropic, so we might not be able to write $L_0$ as a direct sum of $k v_0$ and its orthogonal.) We need to show that this latter Homspace meets ${\mathfrak{so}}(q_0)$ in a subspace of dimension $(d1)(d2)/2$.
When $q_0(v_0) \ne 0$ then $v_0^{\perp}$ is a linear complement to $kv_0$ and moreover $q_0$ has nondegenerate restriction $q'_0$ on this hyperplane, so the intersection in such cases is ${\mathfrak{o}}(q'_0)$, which has the expected dimension (as ${\rm{char}}(k) \ne 2$). Suppose instead that $q_0(v_0) = 0$, so then $B_{q_0}$ defines a perfect pairing $\beta$ between $L_0/kv_0$ and $v_0^{\perp}$, and the $T_0$'s of interest are exactly the skewsymmetric forms relative to this pairing; i.e., $\beta(x,T_0(x)) = 0$ for all $x \in L_0/kv_0$. So in this case the space of interest is $\wedge^2(L_0/kv_0)^{\ast}$, which again has the desired dimension.
QED

$\begingroup$ Thanks you very much for the detailed answer. I am trying to understand it now. $\endgroup$ Jul 13 '14 at 10:05