Edit: after posting the answer below, I noticed the question requires the study of the operator
$$
\int f(y,\alpha)f(y-x,\alpha)\mathrm{d}y\quad\text{ which is not }\quad \int f(y,\alpha)f(x-y,\alpha)\mathrm{d}y.
$$
This implies that the Fourier transform of the right side of \eqref{a} is not the square of the Fourier transform of the unknown, but rather it is the product of its Fourier transform and its inverse Fourier transform. Therefore the application of the Fourier transform does not "algebrize" the nonlinearity, and at the right side, the transformed nonlinear term still involves (the product of two) integral operators.
The development below thus does not provide an answer to the given question: however, since I noticed that it is interesting per se, I asked for advice on Meta and decided to keep it and add this note. An advice
- Read as carefully as possible the question before deciding to work to an answer: at first sight, it may seem something similar to problems you've already been dealing with but it may actually be very different.
I longed a lot working to this answer because I was fooled by a wrong route: however, here it is.
The solution of the Cauchy problem
$$
\left\{
\begin{split}
\frac{\partial}{\partial \alpha} f(x,\alpha)&=\int\! f(y,\alpha)f(x-y,\alpha)\mathrm{d}y\\
\\
f(x,\alpha)|_{\alpha=0}&=f_o(x)
\end{split}\right.\tag{1} \label{1}
$$
exists and is unique even if the Cauchy datum $f_o$ is taken from $\mathscr{E}^\prime(\Bbb R^n)$, $n\ge 1$: if we further assume that $f_o\in C^k_c(\Bbb R^n)$, $k\in\Bbb N$, we have that $f(x,\cdot)\in C^k(\Bbb R^n)$ for every fixed $\alpha\in\Bbb R_+$.
Formal existence and uniqueness of the solution
As suggested by Bob Terrel in his comment, let's start by formally applying the Fourier transform $\mathscr{F}_{x\mapsto\xi}$ to each side of \eqref{1}, for the moment supposing simply $f_o\in \mathscr{E}^\prime(\Bbb R^n)$: this gives us the following, simple ODE
$$
\left\{
\begin{split}
\frac{\mathrm{d}}{\mathrm{d} \alpha} \hat{f}(\xi,\alpha)&= \big(\hat{f}(\xi,\alpha)\big)^{\!2}\\
\\
\hat{f}(\xi,\alpha)|_{\alpha=0}&=\hat{f}_o(\xi)
\end{split}\right.\tag{2} \label{2}
$$
Since $f_o\in \mathscr{E}^\prime(\Bbb R^n)$, its Fourier transform $\hat{f}_o(\xi)$ is the restriction to $i\Bbb R^n$ of the entire function of several complex variables $\hat{f}_o(z)=f\big(e^{-i\langle x,z\rangle}\big)$, $z=\zeta+i\xi\in\Bbb C^n$, i.e. is a complex valued analytic function of the real variable $\xi$ (see for example [2], chapter VII, §7.1, theorem 7.1.14 pp. 165-166): this means that \eqref{2} is a standard Cauchy problem for a classical first order ODE with analytic parameters, and can be solved by using of the standard Barrow's formula ([1], §1.5 p. 19) and the related existence and uniqueness theorem ([1], §2.2 p. 36)
$$
\hat{f}(\xi,\alpha)=
\frac{\hat{f}_o(\xi)}{1-\alpha\hat{f}_o(\xi)} \tag{3}\label{3}
$$
The form of the vector field math the second side of the ODE in \eqref{2} assures that \eqref{3} is its unique solution: still we cannot conclude nothing about the existence and uniqueness of the solution of \eqref{1}, since we don't know if the meromorphic function \eqref{3} (respect to the $\xi$ variable) is the Fourier transform of a distribution (or more generally of a generalized function).
Existence and uniqueness of a solution in $\mathscr{S}^\prime$
The right side of equation \eqref{3} is the product of the Fourier transform of a distribution of compact support and of a meromorphic function which is the solution of the following equation
$$
\big(1-\alpha\hat{f}_o(\xi)\big)v(\xi)=1.\tag{4}\label{4}
$$
Now, equation \eqref{3} can be interpreted as the Fourier transform of the convolution of a distribution of compact support $f_o\in\mathscr{E}^\prime(\Bbb R^n)$ and a tempered distribution $\mathscr{S}^\prime(\Bbb R^n)$, which is always well defined distribution (in this case a tempered one), if and only if the problem \eqref{4} has a solution $v\in\mathscr{S}^\prime(\Bbb R^n)$ whatever the analytic function $1-\alpha\hat{f}_o$ is. But this is exactly a particular instance of the statement of the solution to the division problem, one of the cornerstones of the modern theory of PDEs (and $\psi$DEs), proved by Lars Hörmander in 1958 for the case the analytic function is a polynomial and by Stanisław Łojasiewicz in 1959 for the general case of a real analytic function (and non degenerate systems of such functions, see [3] and [4]). Thus
$$
\hat{f}(\xi,\alpha)\in\mathscr{S}^\prime(\Bbb R^n)\iff \mathscr{F}_{\xi\mapsto x}^{-1}\big(\hat{f}\big)\in\mathscr{S}^\prime(\Bbb R^n)
$$
Thus, by the isomorphism properties of the Fourier transform in $\mathscr{S}^\prime$ (see [2], chapter VII, §7.1, theorem 7.1.10, p. 164), the tempered distribution
$$
f=\mathscr{F}_{\xi\mapsto x}^{-1}\big(\hat{f}\big)=\mathscr{F}_{\xi\mapsto x}^{-1}\Big(\big({1-\alpha\hat{f}_o(\xi)}\big)^{-1}\Big)\ast f_o \tag{5}\label{5}
$$
is the well defined and unique generalized solution of \eqref{1}.
Regularity of the solution
If $f_o\in C^k_c(\Bbb R^n)$, $k\in\Bbb N$, then \eqref{5} becomes the following function:
$$
f(x,\alpha)=\mathscr{F}_{\xi\mapsto x}^{-1}\big(\hat{f}\big)(x,\alpha)
=\mathscr{F}_{\xi\mapsto y}^{-1}\Big(\big({1-\alpha\hat{f}_o(\xi)}\big)^{-1}\Big)\big(f_o(x-\cdot)\big)\tag{5'}\label{6}
$$
and by standard properties of distributions (see for example [5], chapter 3, §3.4 pp. 48-50) $f(x,\cdot)\in C^k(\Bbb R^n)$ for every fixed $\alpha\in\Bbb R_+$.
Notes
- The wrong route to which I alluded above is trying to study \eqref{1} in convolution algebras of various kind: this is due to the fact that, obviously, in order to apply the powerful theorems proved for these objects you must prove that \eqref{3} belongs to one them. This in turn requires a knowledge of the growth behavior in various regions of the complex space, or the absence of zeros and so on. By using the cornerstone solution of the division problem in the general form given by Łojasiewicz, it is possible to get rid of that (a bit cumbersome) machinery.
- If we study the case $n=1$, $f_0\in L^2(K)$, $K$ compact in $\Bbb R$, and consider the convolution in \eqref{1} as a Volterra convolution, i.e.
$$
(f,g)\mapsto f\ast g(x)=\int\limits_0^x f(x-y)g(y)\mathrm{d}y
$$
Then you can proceed in a very simple way, by considering $\big({1-\alpha\hat{f}_o(\xi)}\big)^{-1}$ as the Fourier transform of the resolvent of a convolution type Volterra integral equation of the second kind, which is well defined and always exists. However the hypotheses required severely limit the kind of problems which can be treated in this way.
Bibliography
[1] Vladimir Igorevic Arnol'd, Ordinary differential equations, various editions from MIT Press and from Springer-Verlag, MR1162307, Zbl 0744.34001.
[2] Lars Hörmander (1990), The analysis of linear partial differential operators I, Grundlehren der Mathematischen Wissenschaft, 256 (2nd ed.), Berlin-Heidelberg-New York: Springer-Verlag, ISBN 0-387-52343-X/ 3-540-52343-X, MR1065136, Zbl 0712.35001.
[3] Stanisław Łojasiewicz (1959), "Sur le problème de la division" (French),
Studia Mathematica 18, 87-136, DOI: 10.4064/sm-18-1-87-136, MR0107168, Zbl 0115.10203.
[4] Stanisław Łojasiewicz (1961), Sur le problème de la division, (French), Rozprawy Matematyczne 22, pp. 57, MR0126072, Zbl 0096.32102.
[5] V. S. Vladimirov (2002), Methods of the theory of generalized functions, Analytical Methods and Special Functions, Vol. 6, London–New York: Taylor & Francis, pp. XII+353, ISBN 0-415-27356-0, MR2012831, Zbl 1078.46029.