I believe your set is indeed compact. Its complement is the collection of points $x$ such that there exists a continuous curve $\gamma^x:\mathbb R_+\to M$ such that
- $\gamma^x(0)=x$;
- $\gamma^x$ leaves every compact set;
- the image of $\gamma^x$ stays outside $X$.
According to this point of view, it is clear that this set is open, and decreasing in $X$. It will suffice to show that your set is compact for some $X'\supset X$ large enough.
Let $X'\supset X$ be a compact submanifold with boundary in $M$ of maximal dimension (for instance, embed $M$ as a closed set of $\mathbb R^n$, and consider the intersection with a large close ball; by Sard's lemma it is a manifold with boundary most of the time). I claim that $M\setminus X'$ only has finitely many relatively compact components, which will conclude. Suppose by way of contradiction that it is not the case, so there is a sequenceEvery point of points $(x_n)_{n\geq0}$ belonging to$\partial X'$ admits a compact neighbourhood of $X'$, such that any two of these belong to different relatively compact components of $M\setminus X'$. Up to extraction, we can assume they converge to some $x\in M$. But clearly this cannot be the case: informally, around $x$ there is onlyintersects exactly one connected component of $M\setminus X'$ (sincebecause locally $X'$ is locally just a half-hyperspacespace). By compactness, we can find an open neighbourhood of $\partial X'$ that intersects only finitely many components, and by connectedness of $M$ every such component intersects every neighbourhood of $\partial X'$, so those are finitely many as advertised.