Palais's and Kobayashi's theorems on automorphism groups of geometric structures

Question

My question concerns two results in the neighborhood of the standard theorem of Myers-Steenrod that isometry groups of Riemannian manifolds are Lie groups. Both appear in the first chapter of Kobayashi's book "Transformation Groups in Differential Geometry", and the longer I think about them, the more I doubt whether Kobayashi's exposition can be completely correct. One of them is a theorem due to Kobayashi that can be stated in modern terms as follows:

Theorem 3.2: Suppose $M$ is a smooth manifold with a fixed global trivialization of its tangent bundle, and $G \subset \operatorname{Diff}(M)$ is the group of diffeomorphisms that preserve this trivialization. Then $G$ admits the structure of a Lie group acting smoothly on $M$ such that for any $p \in M$, the map $G \to M : g \mapsto g \cdot p$ sends $G$ diffeomorphically to a smooth submanifold of $M$.

I'm stuck on the final sentence of Kobayashi's proof, so in order to ask my question, I have to summarize the proof. It's based on a theorem of Palais (3.1 in Kobayashi's book), which for simplicity I will state here in the case of a closed manifold:

Theorem 3.1 (closed case): On a smooth closed manifold $M$, suppose $G \subset \operatorname{Diff}(M)$ is a subgroup and $S$ is the set of vector fields on $M$ whose flows belong to $G$. If the Lie algebra generated by $S$ is finite dimensional, then $G$ admits the structure of a Lie group acting smoothly on $M$ such that $S$ is its Lie algebra.

This theorem confused me for a while before I realized there is some sleight-of-hand in the statement: we like to imagine $\operatorname{Diff}(M)$ as carrying a natural topology, but the statement of Palais's theorem doesn't mention any topology on this group, and indeed, the topology produced by the proof might be very different from what you expect. That's because the main step in the proof is to apply the following lemma:

Lemma: Suppose $G$ is a group and $H \subset G$ is a topological group contained in $G$ as a normal subgroup. If the map $H \to H : h \mapsto g h g^{-1}$ is continuous for every $g \in G$, then $G$ admits a unique topology for which $H \subset G$ is an open subset.

In Kobayashi's proof of Theorem 3.1, the subgroup $H \subset G$ is the connected Lie group of diffeomorphisms generated by the flows of vector fields in the set $S$, so he endows $G$ with the topology promised by the lemma, making $H$ the identity component of $G$, and then transfers the smooth structure from $H$ to the other components. I could buy that, except for the following detail:

Question 1: Could the "Lie group" $G$ resulting from Theorem 3.1 have uncountably many connected components?

For instance, one could apply the lemma above to the normal subgroup ${\mathbb R} \subset {\mathbb R}^2$, with ${\mathbb R}$ assumed to carry its usual Lie group structure. The result would be a very strange smooth structure on ${\mathbb R}^2$ in which it is diffeomorphic to a disjoint union of uncountably many copies of ${\mathbb R}$. That is almost certainly not what you want to do in any given situation, and according to what I regard as the standard definitions these days, the result is not a Lie group --- it is not even a manifold, because it is not second countable. But I don't see why such a scenario couldn't happen in Theorem 3.1, and am thus left unsure as to whether the theorem is even true according to standard definitions. (I don't have access to Palais's original proof of this theorem, but I have looked at the exposition in Postnikov's book "Geometry VI: Riemannian Geometry"; Postnikov states the lemma above on page 124 as Exercise 10.3 and does not discuss it any further.)

Now, here is a quick outline of Kobayashi's proof of Theorem 3.2, in which I will again make life slightly easier by assuming $M$ is closed. Recall that $M$ is endowed with a global trivialization of its tangent bundle, so there is a natural notion of "constant" vector fields on $M$.

• Step 1: The map $G \to M : g \mapsto g \cdot p$ is injective.
• Step 2: The $G$-orbit of $p$ is a closed subset of $M$.
• Step 3: The subgroup $H \subset G$ consisting of flows of vector fields is generated by the Lie algebra of all vector fields that commute with the constant vector fields. Moreover, all vector fields in this Lie algebra are nowhere zero.
• Last step: Since the Lie algebra in step 3 is finite dimensional, Theorem 3.1 endows $G$ with a Lie group structure, and the map $G \to M : g \mapsto g \cdot p$ is now an injective immersion with closed image, thus its image is a submanifold of $M$.

Leaving my previous caveat about Theorem 3.1 aside, I was fine until the the very last part, but...

Question 2: It is certainly not true in general that the image of an injective immersion is a submanifold whenever it is closed; so how does Kobayashi's proof actually guarantee that the orbit is a submanifold?

I can imagine various horror scenarios: for instance, if the group in Theorem 3.1 really can end up having uncountably many components, then the map $G \to M : g \mapsto g \cdot p$ might immerse an $(n-1)$-dimensional "manifold" onto an $n$-dimensional neighborhood of $p$. That scenario is not actually so worrying since it then becomes obvious that the orbit is in any case a submanifold of $M$ (namely it is an open subset), but even if $G$ has only countably many components, I can imagine the orbit might be something like a sequence of disjointly embedded connected $(n-1)$-dimensional submanifolds converging to one that passes through $p$, and that would still be a closed image of an injective immersion.

I realize that Kobayashi's theorem also follows from the proof of Myers-Steenrod that isometry groups are Lie groups, and while I have not read the latter in detail, I see no reason to be skeptical of it (it certainly puts a lot more effort into proving that orbits are submanifolds). Of course, I was hoping I wouldn't have to read the rest of Myers-Steenrod because Kobayashi's proof looked (deceptively?) easier.

Answer 1

Question 1: You are absolutely correct that the Lie group structure here is not a priori compatible with the structure of topological subgroup of $Diff(M)$ and, as stated, you can construct examples where your ``Lie group'' is discrete with uncountably many components:

Consider a manifold $M$ with a flow $(\phi_t)$, and an uncountable subgroup $T$ of $\mathbb R$ (maybe you need the axiom of choice for that) and define $G = \{\phi_t, t\in T\}$. Then $G$ should not contain a 1-parameter subgroup, so Theorem 3.1 only endows it with the discrete Lie group structure.

Question 2: In light of the above discussion, the last step of the proof is probably missing some precisions. But here there are two things that prevent bad behaviour as before:

• The group $G$ is closed in $\mathrm{Diff}(M)$
• The group $G$ preserves a Riemannian metric (the metric for which the framing is orthogonal)

By Ascoli, this implies that $G$ is compact (a priori $\mathrm{Homeo}(M)$, but in fact in $\mathrm{Diff}(M)$). Working with this, you can probably fix the proof?

Answer 2

Two weeks after I asked my question, I've decided I should write an answer summarizing the conclusions I've reached in the mean time, based partly on the previous answers and ensuing discussions, plus thinking through how I would try to prove Theorem 3.2 if I'd written the book instead of Kobayashi.

Part 1: Is Kobayashi's proof wrong?

First let's be clear: as I mentioned at the end of my question, there's no actual doubt about Theorem 3.2 being true. One can derive it as an immediate corollary of the theorem of Myers-Steenrod that isometry groups of Riemannian manifolds (and therefore also their closed subgroups) are Lie groups. The question really is just what is the simplest and most elegant way to prove this result. By this measure, I don't find the paper by Myers and Steenrod to be a fantastic candidate, partly because much of its notation and terminology seems painfully dated, and also because it expends quite a lot of effort proving things about finitely-differentiable submanifolds, which seems uglier than what ought to be necessary. A lot of people cite Kobayashi's book instead for this result, and I was hoping to find a simpler proof there.

But at this point I'm fairly convinced that Kobayashi's proof isn't complete, and I'm not all that optimistic that the main idea behind it is salvageable. Referring to the summary I gave in my question, I have no problems at all with steps 1 through 3, but I still have large unanswered questions about the last step -- it is in fact the last sentence of the proof in Kobayashi's book -- which I believe completely skips over what should actually be the difficult part of the proof. From the previous answers to my question, I can piece together a plausible theory of what Kobayashi intended in this step: the finite-dimensional Lie algebra in step 3 determines on $M$ an integral distribution such that the orbit through the point $p \in M$ is a union of integral submanifolds of that distribution. Since we already know that the orbit is a closed set, one would like to deduce from this that the orbit is a submanifold, because in general, connected leaves of foliations are submanifolds whenever they are closed. (That follows from the Baire category theorem, as explained in Tom Goodwillie's answer to this mathoverflow question.)

But the problem we still have is that when we applied Theorem 3.1 in order to call $G$ a Lie group, we gave it a topology which might be very different from any that it would naturally inherit as a subgroup of $\operatorname{Diff}(M)$ or by identifying it with the orbit, and in particular, there is nothing to stop it from having uncountably many connected components. There are good reasons why the modern definition of the word "manifold" typically excludes that possibility, and one of them is that in this situation, the orbit through $p \in M$ might therefore be a union of uncountably many leaves of the foliation, in which case the aforementioned argument via the Baire category theorem does not work and we cannot conclude that it's a submanifold just because it's a closed set.

This looks to me like a big gap that can't be filled in any easy way. The paper by Myers and Steenrod does not have this problem, because they have a direct argument (which takes a few pages) showing that the orbit with its natural topology is a submanifold of $M$. I'm still open to the possibility that I'm missing some simple observation that would fix the last step in Kobayashi's argument, but none of the answers so far have convinced me in this regard.

Part 2: What could a correct proof look like?

Here's an outline of a proof of Theorem 3.2 that I haven't seen in the literature, but I'm pretty sure it works, and it seems to me a bit more elegant than Myers-Steenrod. (It's also not hard to imagine how one might adapt it for things like holomorphic automorphisms on an almost complex manifold, where elliptic regularity is involved.) One caveat is that it requires a completeness assumption about $M$, i.e. that all the vector fields that are constant with respect to the given trivialization have global flows. When applying the result to prove that the isometry group of a Riemannian manifold $(M,g)$ is a Lie group, this amounts to the assumption that $(M,g)$ is geodesically complete. That assumption shouldn't be necessary if one believes Myers-Steenrod, but I haven't figured out how to remove it.

Steps 1-3 in my account of Kobayashi's proof can be adopted without change. In particular, there is a finite-dimensional Lie algebra $L \subset \mathfrak{X}(M)$ consisting of all vector fields on $M$ that commute with all the constant vector fields, and it generates a Lie group $H$ contained in $G$. One important fact in the background is that since we're assuming the constant vector fields are all complete, the vector fields in $L$ are also complete. This is equivalent to the theorem that for any manifold with a geodesically complete affine connection, all infinitesimal affine transformations are complete (see e.g. Theorem II.2.5 in Kobayashi).

The main thing we need to prove is that a neighborhood of the identity in $G$ is contained in $H$. When I say "neighborhood", I am assuming $G$ carries the $C^\infty_{\operatorname{loc}}$-topology, and this also applies to all mentions of convergence of maps or vector fields in the following.

Step 4: Fix the affine connection on $M$ that is trivial with respect to the given trivialization, suppose $\psi_k \in G$ is a sequence converging to the identity, and write $\psi_k = \exp \circ X_k$ for a sequence of vector fields $X_k$ converging to zero. Then after passing to a subsequence, there exists a sequence $\tau_k > 0$ converging to zero such that $$ \frac{1}{\tau_k} X_k \to X_\infty $$ for some nontrivial vector field $X_\infty \in L$. The proof is basically Arzelà-Ascoli, using the fact that the maps in $G$ and vector fields in $L$ all satisfy highly overdetermined first-order PDEs which bound their $m$th derivatives on compact subsets in terms of their $(m-1)$th derivatives.

Actual last step: Arguing by contradiction, suppose $\psi_k = \exp \circ X_k \in G$ is a sequence converging to the identity that cannot be expressed as the time 1 flow of a sequence of vector fields in $L$ converging to zero. Recall from Step 3 that the vector fields in $L$ are nowhere zero, so they span a subbundle $E \subset TM$. Choose a complementary subbundle $E^\perp \subset TM$, and write $X_k = Y_k + Z_k$ where $Y_k$ and $Z_k$ are sections of $E$ and $E^\perp$ respectively. Fix a point $p \in M$, let $Y_k' \in L$ be the unique vector field in $L$ that matches $Y_k(p)$ at $p$, and define $$ f_k \in G $$ as the time 1 flow of $Y_k'$. We can assume after passing to a subsequence that $Z_k(p) \ne 0$ for all $k$, because otherwise $\psi_k(p) = f_k(p)$ and therefore (by Step 1) $\psi_k \equiv f_k$, violating our assumption about $\psi_k$. (This, by the way, is where I'm using the completeness assumption, namely so that $Y_k'$ is guaranteed to have a global flow for defining $f_k$.) Now $f_k^{-1} \circ \psi_k$ is another sequence in $G$ converging to the identity, and writing $f_k^{-1} \circ \psi_k = \exp \circ V_k$, we can pass to a subsequence and find a sequence $\tau_k \to 0$ such that by Step 4, $\frac{1}{\tau_k} V_k$ converges to a nontrivial element $V_\infty \in L$. But the conditions $Y_k'(p) = Y_k(p)$ and $Z_k(p) \ne 0$ imply in this situation that $V_\infty(p)$ is in $E^\perp$, which is a contradiction.

Answer 3

Question 2: The orbit of a Lie group action is always an initial immersed submanifold. See 2.13-2.15, 5.6 and 6.4 in this book for all this.

Added: If an orbit is closed, then it is an embedded submanifold.

Answer 4

I worked out a proof that the automorphism group of a Cartan geometry, on any manifold with finitely many components, is a Lie group, using the topology inherited by the injective mapping of the automorphism group to any of its orbits in the bundle of the geometry (https://arxiv.org/pdf/2302.14457.pdf, Theorem 15). I prove that every bundle orbit is closed and embedded and they are all diffeomorphic (which is also not clear to me in Kobayashi's proof). Since a trivialization of the tangent bundle is a special case (a Cartan geometry modelled on Euclidean space, acting on itself by translation), this fixes up the problem in Kobayashi's proof, which seems to have been known to many people for many years. My previous proof used Myers-Steenrod (as you mentioned), but this proof is much easier. In outline: it follows the usual proof that a closed subgroup of a Lie group is a Lie subgroup.