Are there any connections between continuous dynamical systems and (analytic) number theory?

8$\begingroup$ Continued fractions encode the geodesic flow on the space $PSL_{2}(\mathbb{R})/PSL_{2}(\mathbb{Z})$, so yes. $\endgroup$– AsafOct 2 at 12:21

$\begingroup$ It seems some of the answers given are discretetime, not continuous. $\endgroup$– Jesse ElliottOct 2 at 22:36

1$\begingroup$ @Jesse Elliott. There's some ambiguity with regard to the modifier "continuous". A discretetime dynamical system f^(n) might be regarded as "continuous" if the map f is a continuous selfmap of a topological space. $\endgroup$– Chris JudgeOct 3 at 19:29

1$\begingroup$ To further Asaf's comment: See work of Einsiedler, Katok and Lindenstrauss on Littlewood's conjecture. en.wikipedia.org/wiki/Littlewood_conjecture $\endgroup$– Chris JudgeOct 3 at 19:34
4 Answers
There is a whole area which studies such connections, e.g., revolving around the Sarnak Conjecture: a continuous discretetime dynamical system $(X,T)$ of zero topological entropy, where $X$ is a compact metric space and $T$ is a homeomorphism, must be Möbius disjoint, i.e., orthogonal to the Möbius function from number theory. This means $$ \lim_{N\rightarrow\infty}\frac{1}{N}\sum_{n\le N}f(T^nx) \mu(n)=0 $$ for all $x\in X$ and all real continuous functions $f$ on $X$.
For a review, as of 2017, see "Sarnak's Conjecture  what's new".
As usual in those matters, this is more a long comment than a full comprehensive answer.
I am not aware of works applying directly the theory of dynamical systems to the analytic theory of numbers. Nevertheless I know that the SiegelShidlovskii theory of transcendental numbers relies on the theory of homogeneous linear ordinary differential equations of polynomial, or more generally rational, coefficients which is undoubtedly a chapter of this theory.
It seems that the roots of this theory lie in an ancient observation by A. M. Legendre (according to [1], chapter 1, §4, p. 7 and [2], §4, p. 39): by considering the power series
$$
f_\alpha(x)=\sum_{n=0}^\infty \frac{x^n}{n! \alpha(\alpha+1)\cdots(\alpha+n1)},\quad \alpha\neq 0, 1, 2, \ldots,
$$
which is an entire solution of the following linear ODE with polynomial coefficient
$$
x f^{\prime\prime}(x)+\alpha f^{\prime}(x)= f(x),
$$
Legendre noted that the ratio $f_\alpha(x)/ f^{\prime}_\alpha(x)$ is irrational for any rational $x\neq 0$ and $\alpha$ satisfying the above requirements. Later Erik Stridsberg proved the irrationality of the dividend $f_\alpha(x)$ and of the divisor $f^{\prime}_\alpha(x)$ for the same values of $x$ and $\alpha$, but decisive step in this approach was done by Carl Ludwig Siegel in 1949.
Siegel defined the class of $E$functions by using the following linear system of ODEs with rational coefficients
$$
\frac{\mathrm{d}}{\mathrm{d} x}
\begin{pmatrix}
f_1(x)\\
\vdots\\
f_k(x)\\
\vdots\\
f_m(x)
\end{pmatrix} =
\begin{pmatrix}
Q_{1,1}(x) & Q_{1,2}(x) & \ldots &Q_{1,m}(x) \\
Q_{2,1}(x) & Q_{2,2}(x) & \ldots &Q_{2,m}(x) \\
\vdots & \vdots & \ddots & \vdots \\
Q_{m,1}(x) & Q_{m,2}(x) & \ldots &Q_{m,m}(x)
\end{pmatrix}
\begin{pmatrix}
f_1(x)\\
\vdots\\
f_k(x)\\
\vdots\\
f_m(x)
\end{pmatrix},
$$
and was able to prove, assuming suitable hypotheses on the coefficients and on its solutions $\big(f_1(x), \ldots, f_m(x)\big)$ are verified, that all the $m$ numbers $f_1(\alpha), \ldots, f_m(\alpha)$ where $\alpha$ is and algebraic number which is not $0$ nor it is a pole of the functions $\{Q_{i,j}\}_{1\le i,j\le m}$ are algebraically independent. Later on Andrei Borisovich Shidlovskii, was able to simplify in a meaningful way the hypothesis under which the results of Siegel holds, and thus develop further Siegel's original approach. The two references cited below should be more than sufficient to give a sketch of the theory.
References
[1] Andrei Borisovich Shidlovskii, Transcendental numbers. With a foreword by W. Dale Brownawell. Translated from the Russian by Neal Koblitz. (English) De Gruyter Studies in Mathematics, 12. BerlinNew York: Walter de Gruyter, pp. xix+466 (1989), ISBN:3110115689, MR1033015, Zbl 0689.10043.
[2] Naum Il’ich Fel’dman and Andrei Borisovich Shidlovskii, "The development and present state of the theory of transcendental numbers" (English. Russian original) Russian Mathematical Surveys 22, No. 3, 179 (1967); translation from Uspekhi Matematicheskikh Nauk [N. S.] 22, No. 3(135), 381 (1967), MR214551, Zbl 0178.04801.

1
A number of people including my collaborator, Máté Wierdl, have worked extensively on subsequence ergodic theorems. The idea is that one has an integer sequence $t_1<t_2<\ldots$ and one forms the subsequence ergodic averages $$ \frac 1N \Big(f(T^{t_1}x)+\ldots+f(T^{t_n}x)\Big). $$
If the transformation $T$ is an ergodic measure preserving transformation of a probability space $(X,\mu)$ and $(t_n)$ is just the sequence of natural numbers, the Birkhoff ergodic theorem shows that for $L^1$ functions $f$, the above averages converge, almost surely, to the integral of $f$.
A result of Bourgain established convergence for the sequence of squares for $f\in L^2$. Wierdl showed convergence for $f\in L^p$ ($p>1$) for the sequence of primes. A paper of Boshernitzan, Kolesnik, Wierdl and myself showed that if $t_n=\lfloor g(n)\rfloor$ where $g(n)$ grows at a roughly polynomial rate, but is sufficiently far from rational polynomials, then again there is convergence for $f\in L^p$ ($p>1$).
All of these results rely critically on controlling exponential sums, and therefore analytic number theory.
See Cellarosi and Marklof's paper https://arxiv.org/abs/1501.07661
They are interested in a limit theorem of the expression
$$X_N(t):=\frac{1}{\sqrt N}\sum_{n=1}^{\lfloor Nt\rfloor }e\left(\left(\frac12n^2+\beta n\right)x+\alpha n\right)$$ where $x$ is random and $(\alpha, \beta)\in \mathbb R^2\setminus \mathbb Q^2$. They show that independent of how $x$ is distributed, $X_N$ converges to $\sqrt{t}\Theta_\chi(g\Phi^{2 \log t})$ where $\Phi$ is the geodesic flow on $\widetilde {SL(2,\mathbb R)}\ltimes \mathbb H$ and $g$ is drawn wrt Haar measure.

$\begingroup$ Variable $t$ on the left, $x$ on the right side of the equation? And where does $\chi$ come from? $\endgroup$ Oct 3 at 1:08

2$\begingroup$ @GerryMyerson Fixed the first one and $\Theta_\chi$ is a complicated automorphic function. $\endgroup$ Oct 3 at 1:14