Question

I wonder if the periodic paths of a lightray trapped between two nonconcentric circles, each perfectly reflecting, are known? The behavior of such rays seems chaotically complicated. For example, left below the highlighted initial ray has slope $\frac{1}{12}$, while that to the right has slope $\frac{1}{9}$. (The green circle has radius $\frac{3}{4}$.) After 500 reflections, neither ray has become periodic.
   
I suspect the combination of dispersive and focusing reflection (from the inner and outer circles respectively) leads to this complex behavior. But perhaps the periodic paths are known?

I only know the 2-cycles from rays collinear with the circle centers, and those that Noam Elkies kindly identified in his comment: "regular $n$-gons (and $(n/k)$-gons, i.e. stars) that stay so close to the outer circle that they never hit the inner one."

(Related MO question: "Trapped rays bouncing between two convex bodies.")

---

Update 1. I found one! :-) No doubt among the "simple short cycles" that Noam had in mind:
                

---

Update 2. With the benefit of the search terms helpfully provided by Ian Agol and Igor Riven, I found a useful Physical Review paper by G. Gouesbet, S. Meunier-Guttin-Cluzel, and G. Grehan, "Periodic orbits in Hamiltonian chaos of the annular billiard," Phys. Rev. E 65, 016212 (2001). Their approach is more experimental than theoretical:

Periodic orbits embedded in the phase space are systematically investigated, with a focus on inclusion-touching periodic orbits, up to symmetrical orbits of period 6. Candidates for periodic orbits are detected by investigating grayscale distance charts and, afterward, each candidate is validated (or rejected) by using analytical and/or numerical methods.

The (unstable) 3-cycle I found they label "2(1)1" in their (barely discernable) Fig.5 inventory below:
   

Answer 1

I don't know how to answer your question, but I'll make a reformulation of the question.

Consider the unit tangent bundle to the outer circle, and consider the subset consisting of vectors pointing into the circle. This is an annulus, parameterized by $(\theta,\varphi)\in A=[0,2\pi]\times [0,\pi]/\{ (0,\varphi)\sim (2\pi,\varphi)\}$, where $\theta$ parameterizes the point on the outer circle, and $\varphi$ gives the angle that the unit vector makes with the tangent vector. One may consider the first-return map under the geodesic flow $F: A\to A$ (flow in the direction of the vector until you hit the outer circle again, then reflect). This is a piecewise smooth map. For each $\theta$, there are angles $0< f_1(\theta) < f_2(\theta) < \pi$ such that $F(\theta,\varphi)= (\theta+2\varphi, \varphi)$ for $0\leq \varphi \leq f_1(\theta)$ and $f_2(\theta)\leq \varphi \leq \pi$ (in particular, $F$ is the identity on the boundary of $A$). For fixed $\theta$ and $f_1(\theta) < \varphi < f_2(\theta)$, $F(\theta,\varphi)$ is a more complicated trigonometric function depending on how the geodesic reflects off the inner circle and bounces back to the outer circle, but it has the property that $\varphi$ is increasing, and $\theta$ is decreasing. There is a natural measure on geodesic flow in the plane, the Liouville measure, which restricts to a measure on $A$. Clearly $F$ preserves this measure. I haven't computed the measure, but it is invariant under rotation, so is independent of $\theta$, and is invariant under reflection $(\theta,\varphi)\to (\theta,\pi-\varphi)$. One could reparameterize the $\varphi$ coordinate in terms of the Liouville measure to get an area-preserving homeomorphism of the annulus. So I would suggest you could do a literature search for results on periodic points for area-preserving homeomorphisms of an annulus.

Answer 2

Following up on @Ian's comments: the magic words are "monotone twist maps of the annulus" and "Aubry-Mather theory". googling either of the above (or looking in Katok-Hasselblad, who have a whole chapter on the subject) is your best bet.