Return to Answer

There are many local solutions of this equation. For example, suppose that one starts with a $C^2$ function $f$ on an interval $I\subset\mathbb{R}$ such that $f'$ is positive on $I$ and $f(I)$ is disjoint from $I$. Then an inverse $g:f(I)\to I$ of $f:I\to f(I)$ exists and is $C^2$. Now define $f$ on the interval $f(I)$ so that $f(y) = f''(g(y))$ for $y\in f(I)$. Then for $x\in I$, we will have $x = g(y)$ for some $y\in f(I)$ and, of course, $y = f(x)$. Then $f''(x) = f''(g(y)) = f(y) = f(f(x))$ for all $x\in I$.

Remark: I had a little time to write a draft of my notes on the proofs of the claims I make below and have posted it on my home webpage here. (It would have made a very long post on MO, so I decided that it would be better to just link it to a file in my public directory.)

There are many local solutions of this equation. For example, suppose that one starts with a $C^2$ function $f$ on an interval $I\subset\mathbb{R}$ such that $f'$ is positive on $I$ and $f(I)$ is disjoint from $I$. Then an inverse $g:f(I)\to I$ of $f:I\to f(I)$ exists and is $C^2$. Now define $f$ on the interval $f(I)$ so that $f(y) = f''(g(y))$ for $y\in f(I)$. Then for $x\in I$, we will have $x = g(y)$ for some $y\in f(I)$ and, of course, $y = f(x)$. Then $f''(x) = f''(g(y)) = f(y) = f(f(x))$ for all $x\in I$.

Note 1: For every constant $b\in\mathbb{C}$, there is a unique formal power series with lowest order term $bz$ that satisfies $f''(z) = f(f(z))$. The first few terms are $$ f(z) = bz+\frac{{b}^{2}}{3!}\,{z}^{3} +{\frac {{b}^{3} \left( {b}^{2}{+}1 \right)}{5!}}\,{z}^{5} +{\frac {{b}^{4} \left( {b}^{6}{+}{b}^{4}{+}11\,{b}^{2}{+}1\right)}{7!}}\,{z}^{7}+\cdots.\tag1 $$ When $|b|<1$, this series converges absolutely and uniformly on the disk $|z|^2\le 6\bigl(1{-}|b|\bigr)$, and satisfies $|f(z)|\le |z|$ there. See the Addenedum below for a sharper (but still not sharp) estimate of the radius of convergence.

Though I don't (yet) have a proof, numerical calculations indicate that, when $b$ is a small negative real number, the above function $f$ extends real-analytically and periodically to the entire real line and gives a solution $f:\mathbb{R}\to\mathbb{R}$.

As in the case $a=0$, when $|b|<1$, so that $f$ is a 'formal contraction' on a neighborhood of $a$, it turns out that the series converges absolutely and uniformly on a disc of the form $|z-a| \le r(a,b)$ for some $r(a,b)>0$, so this gives a two-parameter family of local solutions with a contracting fixed point. It remains to be seen whether there are values of $(a,b)$ (other than $(0,0)$) for which the corresponding $f$ extends to an entire holomorphic function on $\mathbb{C}$, or whether there exist nontrivial values of $(a,b)\in\mathbb{R}^2$ for which $f$ extends analytically to a neighborhood of $\mathbb{R}\subset\mathbb{C}$.

Note 1: For every constant $b\in\mathbb{C}$, there is a unique formal power series with lowest order term $bz$ that satisfies $f''(z) = f(f(z))$. The first few terms are $$ f(z) = bz+\frac{{b}^{2}}{3!}\,{z}^{3} +{\frac {{b}^{3} \left( {b}^{2}{+}1 \right)}{5!}}\,{z}^{5} +{\frac {{b}^{4} \left( {b}^{6}{+}{b}^{4}{+}11\,{b}^{2}{+}1\right)}{7!}}\,{z}^{7}+\cdots.\tag1 $$ When $|b|<1$, this series converges absolutely and uniformly on the disk $|z|^2\le 6\bigl(1{-}|b|\bigr)$, and satisfies $|f(z)|\le |z|$ there. See the Addendum below for a sharper (but still not sharp) estimate of the radius of convergence.

Update (1 Mar 2021): One can show that, when $b$ is a small negative real number, the above function $f$ extends real-analytically and periodically to $\mathbb{R}$ and gives a $1$-parameter family of nontrivial solutions $f:\mathbb{R}\to\mathbb{R}$. In particular, such an $f$ extends holomorphically to a strip of fixed width about $\mathbb{R}\subset\mathbb{C}$. (Meanwhile, when $-1<b<0$, the radius of convergence of the power series (1) is only $r(|b|)\in(0,\infty)$ (see the Addendum below), which is a very different behavior from that when $0<b<1$.)

As in the case $a=0$, when $|b|<1$, so that $f$ is a 'formal contraction' on a neighborhood of $a$, it turns out that the series converges absolutely and uniformly on a disc of the form $|z-a| \le r(a,b)$ for some $r(a,b)>0$, so this gives a two-parameter family of local solutions with a contracting fixed point.

Though I don't (yet) have a proof, numerical calculations suggest that, when $b$ is a sufficiently small negative real number, the above function $f$ extends real analytically to the entire real line and gives a solution $f:\mathbb{R}\to\mathbb{R}$.

Though I don't (yet) have a proof, numerical calculations indicate that, when $b$ is a small negative real number, the above function $f$ extends real-analytically and periodically to the entire real line and gives a solution $f:\mathbb{R}\to\mathbb{R}$.