This problem was first considered and solved by Sunada who also coined the term "geodesic random walk", see his 1983 paper Mean-value theorems and ergodicity of certain geodesic random walks. Alas, the authors of the quoted arxiv paper were not aware of this. Any assumptions on curvature and dimension are not necessary - it is just enough to assume that the manifold is compact. As it has been pointed out by Pierre PC, the fact that the Riemannian volume is a stationary measure is an immediate consequence of the Liouville theorem. Its ergodicity with respect to the geodesic random walk is equivalent to the absence of invariant subsets of the manifold, which would follow, for instance, if any two points can be joined by a chain of geodesic segments of length $\delta$. Actually, geodesic random walks are always mixing for sufficiently small $\delta$ - this is a consequence of the fact that the cube of the transition operator has a density which is bounded away from 0 on the diagonal. The latter also implies the uniqueness of the stationary measure.
EDIT I had misattributed the term "geodesic random walk" to Sunada. Actually, it seems to be first introduced in 1975 by Jørgensen The central limit problem for geodesic random walks whose work is quoted by Sunada.