I know that minimal flows are actions for which no proper closed invariant subsets exist, but I am unclear how to understand this concept.
If a coset flow on a quotient space Gamma/S is ergodic, strongly mixing, and minimal, does minimal mean that the orbits are neither periodic nor have an infinite number of periodic orbits?
Your question is about a theorem of Furstenberg.
About the definitions - obviously every periodic orbit is minimal, if exists, hence in the case the action is minimal, you won't have any periodic orbits.
Notice that in the case of homogeneous flows, "usually" (due to measure classification, which is known for many cases (Ratner, Lindenstrauss)), the periodic orbits will be on alg. sub-varieties of lower dimensions, hence will be of zero volume in the coset space.
Notice the interesting case called unique ergodicity, this where you have only one invariant measure. This happens with say irrational rotations on the circle. In general, if you have an amenable group action, you can associate with every orbit an invariant measure by averaging. In the unique ergodicity case, this implies that every point of your space is in fact generic. So you won't have periodic orbits. In particular, this situation implies that the action is minimal.
Here is your example to non-existence of periodic orbits in this situation, due to Furstenberg, which is the theorem in question. Let $\Gamma$ be a uniform lattice inside say $PSL2(R)$, and look at the coset space $\Gamma\G$. Define the horocyclic flow to be $u_{t}=exp(tN)$, where $N$ is the upper-triangular nilpotent matrix with 1 above the diagonal. So this is the correct def. of the horocyclic flow. If you don't want to use exp, then you have the unipotent matrix. You don't exp a unipotent matrix, because the Lie algebra of SL2 is the matrices of trace zero!
Furstenberg's theorem says that the action of the horocyclic flow is uniquely ergodic. Hence every point is generic, and the orbit of every point will be equidistributed. Obviously, this action is strong mixing (in the compact case I described, this is an easy consequence of spectral gap and one can quantify the mixing rate by studying the proper automorphic representations).
In the view of Ratner's theorems, the more general case of this situation is probably given a s.s. group, with an action of a maximal unipotent group, s.t. one does not have any set which is invariant under the unipotent group. Then that system is uniquely ergodic.
P.S. I'm suggesting to read more up to date references, such as the book by Einsiedler & Ward, the proceedings of the Clay Pisa summer school, and maybe Lindenstrauss' own notes about homogeneous flows (I'm not sure if Lindenstrauss' notes are publicly available).
I'm reading R. Ellis, A semigroup associated with a transformation group, Trans. Amer. Math. Soc. 94(1960), 272-281 and trying to grasp the distinction of minimal in the following theorem:
THEOREM: Let G be the three-dimensional connected, simply connected non-compact simple Lie group (that is, let G be the universal covering group of the group of all 2 x 2 real matrices of determinant one), let Gamma be a discrete subgroup of S such that Gamma/S is compact, let X be a non-zero element of the Lie algebra of G whose element is realized as a 2 x 2 real matrix of trace two, and consider the coset flow on Gamma/S induced by the one-parameter subgroup e^Xt of G. Then: • If X has real distinct eigenvalues, then the coset flow on Gamma/S is ergodic, strongly-mixing, and has infinitely many periodic orbits. • If X has real equal eigenvalues, then the coset flow on Gamma/S is ergodic, strongly mixing, and minimal. • If X has imaginary eigenvalues, then the coset flow on Gamma/S is periodic.
