This is an open problem that I learned from Thomas Simon. I will completely understand if the question is judged as non-research level (and it is indeed not related to my research), but I believe a solution would result in some nice, publishable mathematics. The point of this post is to popularise the problem.

We need two definitions.

Definition 1:We say that a function $f$ isbell-shapedon $\mathbb{R}$ if it is infinitely smooth, converges to zero at $\pm \infty$ and the $n$-th derivative of $f$ has $n$ zeroes in $I$ (counting multiplicity).

Definition 2:We say thatthe derivative is interlacing for $f$on $\mathbb{R}$ if $f$ is infinitely smooth and for every $n \geqslant 0$ the zeroes of $f^{(n)}$ and $f^{(n+1)}$ interlace (counting multiplicity).

When we say that the zeroes of $f$ and $g$ interlace, we mean that each connected component of $\{x \in I : f(x) \ne 0\}$ contains exactly one zero of $g$, except perhaps for unbounded components, which are allowed to contain no zero of $g$ — at least when all zeroes of $f$ and $g$ are simple. The definition in the general case is somewhat more involved, but hopefully clear.

Problem:Describe all bell-shaped functions. Describe all functions $f$ such that the derivative is interlacing for $f$.

Natural modifications are allowed. For example:

- $f$ can be assumed to be analytic or entire;
- the problem can be restricted to a finite or semi-infinite interval.

By *description* we mean essentially arbitrary condition that is easier to check than the definition. Ideally, one could hope for a result similar to Bernstein's characterisation of completely monotone functions as Laplace transforms, which can be re-phrased as follows: if $f^{(n)}$ has no zeroes on $\mathbb{R}$, then $f$ is the Laplace transform of either a measure on $[0, \infty)$ (a completely monotone function), or a measure on $(-\infty, 0]$ (a totally monotone function). However, the answer for bell-shaped functions is likely much more complicated.

Easy observations:

- It is clear that $f$ is bell-shaped if and only if it has no zero, it converges to zero at $\pm \infty$ and the derivative is interlacing for $f$.
- By Rolle's theorem, if $f$ has no zeroes and $f$ converges to zero at $\pm \infty$, then $f^{(n)}$ has at least $n$ zeroes.
- The function $\exp(-x^2)$ is bell-shaped: its $n$-th derivative is $P_n(x) \exp(-x^2)$ for a polynomial $P_n$ of degree $n$, so it has no more than $n$ zeroes.
- Similarly, the function $f(x) = (1 + x^2)^{-p}$ is bell-shaped for $p > 0$ (and the derivative is interlacing for $f$ also when $p \in (-\tfrac12, 0)$): the $n$-th derivative of $f$ is of the form $P_n(x) = (1 + x^2)^{-p-n}$ for a polynomial $P_n$ of degree $n$.
- In the same vein, $f(x) = x^{-p} \exp(-x^{-1})$ is bell-shaped on $(0, \infty)$ for any $p > 0$: $f^{(n)}(x) = P_n(x) x^{-p - 2n} \exp(-x^{-1})$ for a polynomial $P_n$ of degree $n$.
- The derivative is interlacing for a polynomial if and only if it only has real roots.
- More generally, the derivative is interlacing for any locally uniform limit of polynomials with only real roots. This class includes $\sin x$, $\exp(x)$, $\exp(-x^2)$, some Bessel functions and many more; see Chapter 5 in Steven Fisk's book.

This topic appeared at least in probability literature, in an erroneous article by W. Gawronski, followed by two articles by T. Simon on stable laws and passage times (with W. Jedidi).

EDIT:

Having read through Widder and Hirschman's book and other references pointed out in Alexandre Eremenko's answer (which was really helpful!), I was able to extend Thomas Simon's work and prove that certain functions are bell-shaped. The preprint is available at arXiv; all comments welcome.

The class of functions that I was able to handle includes density functions of infinitely divisible distributions with Lévy measure of the form $\nu(x) dx$, where $x \nu(x)$ and $x \nu(-x)$ are completely monotone on $(0, \infty)$. Such functions are called *extended generalised gamma convolutions* (EGGC) by L. Bondesson, and it includes all stable distributions (thus correcting an error in W. Gawronski's article).

Of course this does not answer the question about characterisation of *all* bell-shaped functions: this remains an open problem.

However, I am not aware of any bell-shaped function outside this class. Is there any?

It allows one to show that certain simple functions are bell-shaped. For example, if I did not make a mistake, $f(x) = \dfrac{1}{(1 + x^2)(4 + x^2)}$ is an EGGC, so it is bell-shaped. Out of curiosity: is there any elementary way to prove that this particular function $f$ is bell-shaped?