I got interested in a similar problem (not quite the same thing, though) a few years back but could not find anything really interesting. Here is what I found in the litterature;
- In case $X$ is compact and $G$ is abelian then there is an invariant metric (compatible with the topology, of course) under the action of $G$ iff $G$ is relatively compact in the homeomorphism group of $X$ (endowed with the compact-open convergence topology) iff the action of $G$ is topologically equicontinuous.
By a result of Marjanovic (" On topological isometries", Indag. Math. 31(1969), 184–189) this extends to the case when $X$ is locally compact (that is not the way Marjanovic's result is formulated but it is equivalent)
"Topologically equicontinuous" means the following: for any $x,y \in X$, any open set $V$ containing $y$, there is an open set $U$ containing $x$ and an open subset $W$ containing $y$ and contained in $V$ such that for all $g \in G$
$g(U) \cap W \neq \emptyset \Rightarrow g(U) \subseteq V$
- In case $X$ is not locally compact then some extensions of this result are known. I know of two papers on this:
C. Borges, How to recognize homeomorphisms and isometries, Pacific Journal
of Mathematics 37(3) (1971), 625–633.
M. Tak Kiang, On some semigroups of mappings, Indag. Math. 33(1972),
I have not found anything in the litterature more recent than that, but I probably haven't looked hard enough - this was more curiosity than serious research on my part (update: well now I have looked more seriously and still haven't found anything else in the litterature).
I did obtain the following: assume $G$ is abelian (this is the only case I thought about, as I was mostly interested in the case when $G$ is generated by $1$ element), that $X$ is Polish and that the action of $G$ on $X$ is topologically transitive. Then there is a compatible $G$-invariant metric if, and only if, $G$ is a topologically equicontinuous group of homeomorphisms of $X$.
Under the same assumptions, there is a complete invariant metric iff any invariant metric is complete.
In case you can read French there are some notes on this on my webpage - I wrote them for myelf so probably you should not take anything that's written on faith...
UPDATE (July 4, 2014): I corrected some imprecisions in the text above (about the locally compact case). I logged in to mention that I. Ben Yaacov and I recently worked on this problem again, and proved the following result (there is a preprint on my webpage which supersedes the notes alluded to above):
Assume $X$ is separable metrizable and $G$ is a group of homeomorphisms of $X$. Then there exists a compatible $G$-invariant metric on $X$ if and only if, for any $y \in X$ and any open $V$ containing $y$ there exists an open $W$ containing $y$ and contained in $V$ such that for any $x \in X$ there exists an open $U$ containing $x$ and satisfying
$\forall g \in G \ (gU \cap W \ne \emptyset) \Rightarrow gU \subseteq V$.
That is a lot of quantifiers! I think they are really needed; the property above is a uniform version of topological equicontinuity (obtained by switching a universal and an existential quantifier).