As everyone knows if $x\in S^1$, then the set $\{ x^n \}$ is either finite or dense. Under which condition is true for any other locally compact group, i.e if $G$ is a locally compact group, and $x\in G$ is a nontrivial element, then the set $x^n$ is either finite or dense.

It is easy to produce groups $G$ in which the set $\{x^n\}$ is finite for every $x\in G$. E.g, let $F$ be any finite group; then $G=F^{\mathbb{N}}$ is a compact, totally disconnected group with this property. So in the OP we may assume that $G$ contains some element generating an infinite cyclic dense subgroup. I claim that, in this case, $G$ is indeed isomorphic to $S^1$. First, $G$ is locally compact abelian. Here is a result I found in the book by W. Rudin, "Fourier analysis on groups" (Wiley, 1962). Say that a locally compact abelian group is {\it monothetic} if it contains a dense cyclic subgroup. Theorem 2.3.2: every monothetic group is either compact or isomorphic to $\mathbb{Z}$ (as a topological group). Coming back to the OP, our group $G$ is monothetic. By the theorem just quoted, $G$ must be compact (as $\mathbb{Z}$ clearly does not satisfy the assumptions of the OP). Let $\hat{G}$ be the dual of $G$, a discrete, infinite abelian group. By Pontrygin duality, it is enough to prove that $\hat{G}$ is infinite cyclic. Dualizing the assumptions in the OP, we see that every homomorphism $\hat{G}\rightarrow S^1$ either has finite image, or is injective. In particular, $\hat{G}$ is just infinite (infinite, but every proper quotient is finite). It is not very difficult to see that a just infinite, abelian group is infinite cyclic (see McCarthy, Donald, Infinite groups whose proper quotient groups are finite. I. Comm. Pure Appl. Math. 21 1968 545–562). 


I don't think ever for a Lie group $G$. If $G$ is not compact then it will have a closed subgroup isomorphic to $\mathbb{R}$, which rules it out. If it is compact, then it will have a proper subgroup isomorphic to $S^1$ which again rules it out. 


This is not true even for compact abelian groups. If you take $G = S^1 \times S^1$ and $(x,y) \in G$ is an element such that the orbit of $x$ in $S^1$ is dense and the orbit of $y$ in $S^1$ is finite, then the orbit of $(x,y)$ in $S^1 \times S^1$ will be neither. 


For Lie groups, the only one with this property is $S^1$. To see this, consider a one parameter subgroup given by exponential of a vector in the lie algebra, and the closure of this subgroup should be abelian. Since the only abelian lie group with the property you are asking is $S^1$, this concludes. For other groups, I believe it should also be true, but I don't know. The argument above, shows that if the group is not abelian, then it does not hold (since the closure of an orbit is an abelian closed subgroup), but I don't know if abelian topological groups are all known (the ones I know, the only one verifying your property is $S^1$). 

