In general, no, even if $M$ and $X$ are $C^\infty$.
By the Whitney extension theorem, one can construct a non-negative $C^\infty$ function $f$ on $\mathbb{R}^d$ with compact support a non-smooth closed nbd of the point $p$ (e.g., in dimension 2, a Koch curve). Also, such that $p$ is a maximum for $f$, and the only critical point of $f$ in $U:=\{f >0\}$. In such a situation, $U$ is exactly the unstable manifold of $p$ for the gradient field $X$ of $f$. Note that since $f$ has compact support, you can transport it on any manifold $M$.