How to solve $f(f(x)) = \cos(x)$? I found the following equation on some web page I cannot remember, and found it interesting:
$$f(f(x))=\cos(x)$$
Out of curiosity I tried to solve it, but realized that I do not have a clue how to approach such an iterative equation except for trial and error. I also realized that the solution might not be unique, from the solution of a simpler problem
$$f(f(x)) = x$$
which has, for example, solutions $f(x) = x$ and $f(x) = \frac{x+1}{x-1}$.
Is there a general solution strategy to equations of this kind? Can you perhaps point me to some literature about these kind of equations? And what is the solution for $f(f(x))=\cos(x)$ ?
 A: The half-iterate of a function can be found by expressing its superfunction in a form of Newton series:
$$f^{[1/2]}(x)=\sum_{m=0}^{\infty} \binom {1/2}m \sum_{k=0}^m\binom mk(-1)^{m-k}f^{[k]}(x)$$
Where $f^{[k]}(x)$ means k-th iterate of $f(x)$
This series converges if two criteria are met:

*

*The superfunction(flow) of f(x) grows not faster than an exponent


*Runge phenomenon does not appear.
There is a number of strategies to combat Runge phenomenon which are outside of this answer's scope. It is worth noting though that trying to find a half iterate of the function $f(x)=\cos x$ leads to this Runge swamp and one needs to employ one of the mentioned techniques to achieve convergence.
Opposite case is with the function $f(x)=\sin x$. The superfunction is limited by $\pm 1$ and the series converges without any problem.
Below is a plot of half-iterate of $\sin x$, obtained with this formula. It is periodic with the same period as $\sin x$. The blue curve is the half-iterate, and the red curve is the half-iterate, repeated twice, and we can see that it is indeed very similar to sine function.

This plot is made from the first 50 terms of the above series.
This formula for the half-iterate can be used to find not only half-itertes but any real (or even complex!) iterate of a function by substituting the needed value instead of 1/2.
The formula can be also written in the following forms:
$$f^{[s]}(x)=\lim_{n\to\infty}\binom sn\sum_{k=0}^n\frac{s-n}{s-k}\binom nk(-1)^{n-k}f^{[k]}(x)$$
$$f^{[s]}(x)=\lim_{n\to\infty}\frac{\sum_{k=0}^{n} \frac{(-1)^k f^{[k]}(x)}{(s-k)k!(n-k)!}}{\sum_{k=0}^{n} \frac{(-1)^k }{(s-k) k!(n-k)!}}$$
There are also some other formulas giving the same result.
A: This does not directly answer the question, which concerned iterative roots on the real axis. However, I have also seen complex solutions mentioned in some of the discussions, so it seems relevant to mention the following.
Baker (in "The iteration of entire transcendental functions and the solution of the functional equation f{f(z)} = F(z)") proves the following.
Theorem 1. If $F(z)$ is an entire function of finite order bounded on some continuous curve $\Gamma$ which extends to infinity, then the functional equation $f\{f(z)\} = F(z)$ has no entire solution.
Recall that finite order means that $\log_+ \log_+ |F(z)| = O(\log|z|)$ as $z\to\infty$. Of course, $\cos$ has order $1$, and is bounded on the real axis. So the equation $f(f(z)) = \cos(z)$ has no analytic solution defined on the entire complex plane. 
Baker's argument is beautifully simple: Firstly, it is well-known that the composition of two functions of positive order cannot have finite order, so any such solution $f$ would have order zero. Now, either $f$ is itself bounded on $\Gamma$, or $f(\Gamma)$ is unbounded and $f$ is bounded on $f(\Gamma)$. In either case, $f$ is bounded on an unbounded connected set. But it is well-known that this is impossible for functions of order less than $1/2$. 
A: There are no continuous solutions. Since the cosine has a unique fixed point $x_0$ (such that $\cos x_0=x_0$), it should be a fixed point of $f$. And f should be injective and hence monotone (increasing or decreasing) in a neighborhood of $x_0$. Then $f(f(x))$ is increasing in a (possibly smaller) neighborhood of $x_0$ while the cosine is not.
As for discontinuous ones, there are terribly many of them ($2^{\mathbb R}$) and you probably cannot parametrize them in any reasonable way. You can describe them in terms of orbits of iterations of $\cos x$, but I doubt this would count as a solution of the equation.
UPDATE: Here is how to construct a solution (this is technical and I might overlook something).
Let X be an infinite set and $g:X\to X$ is a map, I am looking for a sufficient conditions for the existence of a solution of $f\circ f=g$. Define the following equivalence relation on $X$: $x$ and $y$ are equivalent iff $g^n(x)=g^m(y)$ for some positive integers $m$ and $n$. Equivalence classes will be referred to as orbits (the term is wrong but I don't know what is a correct one). Two orbits are said to be similar is there is a bijection between them commuting with $g$. If $Y$ and $Z$ are two similar orbits, one can define $f$ on $Y\cup Z$ as follows: on $Y$, $f$ is that bijection to $Z$, and on $Z$, $f$ is the inverse bijection composed with $g$.
So if the orbits can be split into pairs of similar ones, we have a desired $f$. Now remove from the real line the fixed point of cos and all its roots ($\pi/2$ and the like). Then, if I am not missing something, in the remaining set $X$ all orbits of $\cos$ are similar, so we can define $f$ as above. Define $f$ so that $0$ has a nonempty pre-image (that is, the orbit containing $0$ should be used as $Z$ and not as $Y$). Finally, map the fixed point of $\cos$ to itself, and the roots of $\cos$ to some pre-image of $0$.
A: About literature related to the topic of this question:
In the answer to the question f(f(x))=exp(x)-1 and other functions “just in the middle” between linear and exponential. one can find an interesting link with many references related to the problem:


*

*Lars Kindermann: Iterative Roots and
Fractional Iteration
There is also Kindermanns PhD thesis about finding solutions to iterative functional equations using a neural network (in german only)


*

*Kindermann(2001): Neuronale Netze zur Berechnung
Iterativer Wurzeln und Fraktionaler
Iterationen
which might be helpful.
A: Joel's answer made me think a bit and I believe I found an interesting solution for $f(x)$ :
$$ f(x) = \begin{cases} ix & \text{if } \mathrm{Im}(x) = 0, x\neq 0 \\ \cos(ix) & \text{if } \mathrm{Re}(x) = 0,x \neq 0 \\ 2\pi i & \text{if } x = 0 \end{cases}$$
It is of course a bit of a trick (reminds me of Wick Rotation), but I it works for all $x \in \mathbb R$, because
$$f(f(x)) = \cos(i(ix))=\cos(-x) = \cos(x)$$
Update: Added the case $x=0$. For this we have
$$f(f(0)) = \cos(i(2\pi i))=\cos(-2\pi) = \cos(0)$$
A: If you assume, that $f$ can be written as a power series, lets say 
$f(x)=\sum_i a_ix^i$ (and the power series converges everywhere absolutely), then one can write down a power series for $f\circ f$, where the i-th coefficient is a polynomial in the lower coefficients.
Comparing the coefficients of this power series to the one of $cos$, we have to solve a system of algebraic equations (which might lead to the disambiguity).
This is just a first idea from a non expert.
A: Henrik's idea is good, but it doesn't quite work for $\cos x$ since the power series must have a non-zero leading term, and so can't be substituted into a power series. To fix this, consider a power series about a root of $\cos x$, for example $f(x) = \sum_i a_i (x-\pi/2)^i$ with $a_0=0$.
A: There are a truly enormous number of solutions, if one only wants the solution to work on an interval. Indeed, one can find solutions to $f(f(x)) = g(x)$ for any function $g$ defined on an interval.
Specifically, I claim that if $g:[a,b] \to \Bbb R$, then there are
$2^{|\Bbb R|}$ many functions $f$ from $\Bbb R$ to $\Bbb R$ with
$f(f(x)) = g(x)$ for all $x$ in $[a,b]$.
One such solution $f$ is obtained as follows. First choose a
$z$ such that $[a,b]$ and $[a + z, b + z]$
are disjoint. Now let $f(x) = x + z$, for $x$ in $[a,b]$, and
$f(x) = g(x - z)$, for $x$ in $[a + z, b + z]$. Thus, $f(x)$ first
translates $x$ to another interval, when $x$ is in $[a,b]$, and
then $f$ computes $g$ of the reverse translate, when $x$ is not
in $[a,b]$. So $f(f(x)) = g(x)$.
When $g$ is continuous, then this function $f$
will be continuous also, and can be made total by linearly
extending.
More generally, if $h$ is bijection of $[a,b]$ with another
interval $[a',b']$ disjoint from $[a,b]$, then let $f(x) = h(x)$
for $x$ in $[a,b]$, and $f(x) = g\big(h^{-1}(x)\big)$ for $x$ in
$[a',b']$. It follows that $f(f(x)) = g(x)$. And since there
are $2^{|\Bbb R|}$ many such functions $h$, there are
similarly many functions $f$ satisfying the equation.
A: I like this question so I thought I'd bump it and give my two cents.
We'll denote $I$ as the immediate basin of the fixed point $x_0$ on the real positive line. This is the largest connected set about $x_0$ wherein $\lim_{n\to\infty} \cos^{\circ n}(x) \to x_0$. And we'll denote $\Psi$ as the Schroder function of $\cos$ about $x_0$--the function which linearizes $\cos$ i.e: $\Psi(\cos(x)) = -\sin(x_0)\Psi(x)$.
Naturally in a neighborhood of $x_0$ there are two half iterates
$$f_{01}(x) = \Psi^{-1}(\sqrt{-\sin(x_0)}\Psi(x))$$
for both branches of $\sqrt{}$ ($f_0$ and $f_1$ denoting different branches.). Now, these functions can be lifted to functions $f_{01} : I \to I$. The formula is a little cumbersome but essentially is the following (the proof is exhausting and my own so I'll leave it out). 
Define:
$$\vartheta(x,t) = \sum_{n=0}^\infty \cos^{\circ 2(n+1)}(x)\frac{t^n}{n!}$$
and for $0 < \Re(z) < 1$
$$\phi(x,z) = \frac{1}{\Gamma(1-z)}\int_0^\infty \vartheta(x,-t)t^{-z}\,dt$$
which satisfies $\phi: I\times\mathbb{C}_{0 < \Re(z) < 1}   \to I$ in $x$ and locally about $x_0$ looks like
$$\phi(x,z) = \Psi^{-1}(\sin(x_0)^{2z}\Psi(x))$$
choose $z_0$ and $z_1$ so that $\sin(x_0)^{2z_{01}} = \sqrt{-\sin(x_0)}$ for each branch of $\sqrt{}$ and voila $\phi(z_0,x) = f_0(x)$ and $\phi(z_1,x) = f_1(x)$. We've analytically continued $f_0$ and $f_1$ to the immediate basin. These are also the only analytic solutions to the equation
$$g : I \to I$$
$$g(g(x)) = \cos(x)$$
They are NOT real to real, which is all to do with the multiplier $-\sin(x_0)$ which is negative; there are no real square roots of negative numbers = there are no real composite square roots of functions with negative multipliers. Similarly for the function $h(h(h(x))) = \cos(x)$ there are 3 solutions, and only one of them is real to real (just like there are three cube roots of negative one and only one of them is real). Just as well, there are four solutions to $q(q(q(q(x)))) = \cos(x)$ and none are real to real (there are no real fourth roots of $(-1)$). So on and so forth.
Now I don't have a rigorous proof of the following, but it seems obvious enough that $f_0$ and $f_1$ are defined on their maximal domain. $\partial I$ is part of the julia set, and the function $\vartheta(x,t)$ diverges on the julia set  because $\cos^{\circ 2(n+1)}(x)$ grows super exponentially with $n$ and the factorial no longer does its job. I think this is good intuition in inferring that $f$ has no extension to a larger domain. I could be wrong though--it'd be nice to see $f_{01} : \mathbb{C} \to \mathbb{C}$.
A: Near a fixed point of $\cos(x)$ use the method of Schröder (1871) ...
http://en.wikipedia.org/wiki/Schr%C3%B6der%27s_equation
Best historical reference: Daniel S. Alexander, A History of Complex Dynamics from Schröder to Fatou and Julia (1994).
A: It seems that contrary to some other answers a continuous solution can be constructed.
First of all we interpolate with Newton series the flow of function $\cos(\cos(z))$:
$$\phi_{1/2}(x,z)=\cases {
  \arccos^{[x]}(z), & \text{if } x < 0 \cr
  \cos^{[x]}(z),  & \text{if } x \ge 0 }
$$
$$\phi_{1}(x,z)=\sum_{m=0}^\infty \binom{x/2+1}{m} \sum_{k=0}^m (-1)^{k-m} \binom{m}{k} \phi_{1/2}(k-1,z)$$
We interpolate from the first integer point where the value is real, i.e. from x=-1.
We now obtain the approximation of the other half-flow of cos x by taking arccos on the above function:
$$\phi_{2}(x,z)=\arccos(\phi_{1}(x+1,z))$$
We know that the flow of cos(x) should coincide with the first function in even integers and with the second function in odd integers.
So we make a stub of the flow following this knowledge (we also want that its absolute value to be monotonous).
$$\phi(x,z)=\frac{1}{2} \left((-1)^{x+1}+1\right) (\phi_{1}(x,z)-\text{FP})+\frac{1}{2} \left((-1)^x+1\right) (\phi_{2}(x,z)-\text{FP})+\text{FP}$$
where FP is the cosine fixed point.
This function coincides with the flow in integer points but still disagrees in between.
To get a real flow we have to take a limit of repeated arccosine on the our stub:
$$\Phi(x,z)=\lim_{n\to\infty}  \arccos^{[n]} (\phi(x+n,z))$$
Numerically this limit converges quite fast. If the limit exists, it by definition, satisfies the equation
$$ \cos(\Phi(x,z))=\Phi(x+1,z)$$
so it is the true flow.
The above can be illustrated by the graphic:

Here upper semi-flow (flow of cos(cos z)) ) is blue, lower semi-flow is red, real part of the flow is yellow, imaginable part of the flow is green. All flows are taken as point z=1.
Following this we can build a graphic of half-iterate of cosine $\Phi(1/2,z)$:

Here blue is the real part and red is the imaginary part.
We can verify that the half-iterate repeated twice $\Phi(1/2,\Phi(1/2,z))$ (blue) follows cosine (red) quite well at positive half-periods, and anywhere the cone is positive (that is, on the imaginary axis as well):

I think this coincodes with the answer by Gerald Edgar above.
A modified function, iterated twice gives cosine in all real axis:

This is a true half-iterate of cosine, which works on the whole real axis, producing exactly cosine:

But as has been noted by Joel David Hamkins above, there is infinite number of such solutions, none of which work for the whole complex numbers.
This function can be considered though as the true solution on the complex plane if interpreted as a multi-valued function. To do this, take the function on the each interval and analytically extend it to the whole complex plane.
A mathematica notebook that produces the above is as follows:
$PlotTheme = None; 
f[x_, z_] := If[x >= 0, Nest[Cos, z, 2*x], Nest[ArcCos, z, -2*x]]
n := 30
s := 15
Ni[x_, z_] := 
 Sum[Binomial[x + 1, m]*
   Sum[(-1)^(k - m)*Binomial[m, k]*f[k - 1, z], {k, 0, m}], {m, 0, n}]
Semi2[x_, z_] := Ni[x/2, z]
Semi1[x_, z_] := ArcCos[Semi2[x + 1, z]]
FP := Evaluate[N[FixedPoint[Cos, 1.]]]
a := 21
Flow2[x_, z_] := 
 FP + (Semi2[x, z] - FP)*(((-1)^x + 1)/2) + (Semi1[x, z] - 
     FP)*(((-1)^(x + 1) + 1)/2)
FL[x_, z_] := Nest[ArcCos, Flow2[x + a, z], a]
Plot[{Semi1[x, 1], Semi2[x, 1], Re[FL[x, 1]], Im[FL[x, 1]]}, {x, -5, 
  5}, AspectRatio -> Automatic, PlotRange -> 3]
Plot[{Re[FL[0.5, x]], Im[FL[0.5, x]]}, {x, -5, 5}, 
 AspectRatio -> Automatic, PlotRange -> 3]
Plot[{Re[FL[0.5, FL[0.5, x]]], Cos[x]}, {x, -5, 5}, 
 AspectRatio -> Automatic, PlotRange -> 3]
HalfCos[z_] := 
 If[Im[z] == 0, Sign[Re[Cos[z]]]*FL[0.5, z], Sign[Re[z]]*FL[0.5, z]]
Plot[{Re[HalfCos[x]], Im[HalfCos[x]]}, {x, -5, 5}, 
 AspectRatio -> Automatic, PlotRange -> 3]
Plot[{Re[HalfCos[HalfCos[x]]], Cos[x]}, {x, -5, 5}, 
 AspectRatio -> Automatic, PlotRange -> 3]

