I have a general question in Riemannian geometry: Let M be a compact manifold and $\partial M \neq \emptyset$. Then shoot a geodesic from any boundary point perpendicularly into the interior of M. How can one prove it will end at boundary? If so, it induces a transformation of $\partial M$, does anyone know any result about this transformation? For example, one can ask rigidity property, i.e. if the transformation is an isometry, can one say anything about M?

It doesn't seem true to me that the geodesic will always return to the boundary. It may end up accumulating around a closed geodesic in the interior. For example, consider the hyperboloid of one sheet $$x^2+y^2z^2=1$$ in $\mathbb{R}^3$. This is a surface of revolution and so one can talk of the angular momentum of a path with respect to the axis. For a geodesic this must be constant (Clairault's theorem). If a geodesic has momentum exactly one (in some wellchosen units!) it will accumulate on the "waist" of the surface, where $z=0$. Of course this is a noncompact example, but if a geodesic accumulates on the waist, it stays inside some compact region. So it is easy to imagine a compact surface with boundary which contains the relevant compact region and for which this accumulating geodesic meets the boundary at a right angle. This gives an example where the geodesic doesn't return to the boundary. (I think it's even possible to find an example of a metric on the disc of this sort.) (In case you're curious, a geodesic on the hyperboloid starting with $z>0$, pointing "down" and with momentum less than 1 will pass through the waist and end up asymptotic to a meridian at z = infinity. If the momentum is bigger than 1 it will not reach the waist and swing back upwards, ending up asymptotic to a meridian at z = +infinity.) 


In fact it is not true that the geodesic will shoot again the boundary. You can already construct such a contexample on a disc. First thing that you should do is to understand that if we have a hyperbolic annulus i.e. H^2/Z it has one closed geodesics and there are other (inifinite) geodesics that spriral and finally tend to the closed one. Such a geodesic is contained in a half of the cylinder. Now you can make a cut on the cylinder that will intersect the geondesic by the angle pi/ end through the rest. Finally replace the other infinite half cylinder by a disk. There are some cases when this map is a piescewise isomtetry  there is a whole science about this topic, called interval exchange transformations. This happen for surfaces of any genus >1. It is not clear how much you can say about M if the map is well defined. For example you can take MxI  a direct product. But in the case when the boundary of M is connetced I don't see for the moment any other example than just a standard disk D^n (say with metric of constant curvature) 


You can also see this indirectly in some examples: Take a hyperbolic surface $S$ with a single geodesic boundary component $\gamma$. Choosing inward pointing normals to $\gamma$, you can lift $\gamma$ to the unit tangent bundle. If all paths perpendicular to $\partial S$ ended in $\partial S$, the geodesic flow (suitably interpreted) would produce a nontrivial homotopy from $\partial S$ into $\partial S$. This would produce a nontrivial conjugacy $t\gamma t^{1} = \gamma^k$ in the fundamental group, which can't happen. 

