Boundary of unstable manifold

Question

Let $X$ be a vector field on a compact manifold $M$ that has the form $$ X = \lambda_1 x^1 \partial_1 + \dots + \lambda_n x^n \partial_n + \dots$$ with respect to some chart $x$ around a point $p$. Also, let $\lambda_1, \dots, \lambda_n > 0$.

By the stable manifold theorem, there is an $n$-dimensional unstable submanifold $N$ of $M$ around $p$, i.e. a manifold with $\lim_{t \rightarrow -\infty} \Phi_t(p^\prime) = p$ for each $p^\prime \in N$, where $\Phi_t$ is the flow of $X$.

Thus, $N$ is just an open set in $M$. My question is: Is the boundary of $N$ smooth (I suspect yes) and if so, how to prove it or where to look it up?

Answer 1

In general, no, even if $M$ and $X$ are $C^\infty$.

A simple example on a flat torus $M$. The function $f(x,y):=\cos(x)\cos(y)$ has a non-degenerate maximum point at the origin. The unstable manifold of the origin wrto the gradient field of $f$ is the square $(-\pi,\pi)\times(-\pi,\pi)$, not smooth.

More generally, by the Whitney extension theorem, one can construct a non-negative $C^\infty$ function $f$ on $\mathbb{R}^d$ with compact support a non-smooth closed nbd of the point $p$ (e.g., in dimension 2, a von Koch curve). Also, such that $p$ is a maximum for $f$, and the only critical point of $f$ in $U:=\{f >0\}$. In such a situation, $U$ is exactly the unstable manifold of $p$ for the gradient field $X$ of $f$. Note that since $f$ has compact support, you can transport it on any manifold $M$.