Your setup defines set-valued dynamics on $X$. More precisely, you have
$$X \stackrel{p}{\leftarrow} \overline{G} \stackrel{q}{\rightarrow} X$$
where $p$ and $q$ are the obvious projection maps. The set-valued transformation in question sends $x \in X$ to the subset $qp^{-1}(x) \subset X$. I'll write $F = qp^{-1}$ for convenience, noting that $F:X \to 2^X$
Now, the set $I \subset X$ is invariant in your sense precisely when $F(I) \subset I$. There is a ton of literature on set-valued dynamics, and I'm certainly in no position to do the field justice. My favorite reference is Ethan Akin's work: he has an entire book called "the general topology of dynamical systems" which would be relevant. On his webpage you can find a ton of references: I'd start with the "Tourist's Guide" survey paper, it is extremely well-written.
If you don't care much about the precise structure invariant sets sitting inside some compact subset $A \subset X$, you can use the Conley index theory to get sufficient conditions for the existence of a (nonempty) maximal invariant set $I \subset A$ under the (minor) additional assumption that $A$ isolates $I$. Briefly, define the exit set $E \subset A$ to be the collection of those $a \in A$ whose forward orbit sets $F^n(a)$ eventually leave $A$. Now, if the homotopy type of the quotient $A/E$ (called the homotopy conley index) is non-trivial (again, we need some niceness properties on $A$ and $E$, maybe an ANR pair or something) then you are guaranteed a non-empty invariant set $I \subset A$. Sadly, this is not if and only if: it is possible to have invariant sets inside $A$ with trivial Conley index.