Let $u \in C^\infty_c(\mathbb{\Omega})$ and $\varphi$ be an eigenfunction of the fractional Laplacian $(\Delta)^s$ in $\Omega$ with eigenvalue $\lambda$. In what sense, if any, is it true that $$\langle u, \varphi \rangle = \frac{1}{\lambda} \langle u, (\Delta)^s\varphi \rangle = \frac{1}{\lambda} \langle(\Delta)^s u, \varphi \rangle = \frac{1}{\lambda^2} \langle (\Delta)^{2s} u, \varphi \rangle = ........?$$ Here $ \langle \cdot, \cdot \rangle$ denotes the scalar product in $L^2(\Omega)$. In other words, is it true that we can integrate by parts using $(\Delta)^su$, $(\Delta)^{2s}u$ and so on even though their support is not compact anymore?
1 Answer
No, we cannot.
Formally, $\varphi$ is the eigenfunction of the unbounded operator $L_s$ on $L^2(\Omega)$, defined initially by $$ L_s u(x) = (\Delta)^s u(x) \qquad \text{for } x \in \Omega , $$ where $u \in C_c^\infty(\Omega)$ (and it is understood that $u(x) = 0$ for $x \notin \Omega$), and then extended to an appropriate domain (e.g. by means of Friedrichs extension).
Now the key observation is that $L_s L_s$ is not equal to $L_{2s}$, unless $\Omega = \mathbb R^d$. Indeed, in $\Omega$ we have $$ L_s L_s u = (\Delta)^s (\mathbb 1_{\Omega} \times (\Delta)^s u) , $$ while $$ L_{2s} u = (\Delta)^s (\Delta)^s u , $$ and due to nonlocality of $(\Delta)^s$, the two are not equal.
For the above reasons, we have $$ \langle u, \varphi \rangle = \frac{1}{\lambda^2} \langle u, L_s L_s \varphi \rangle = \frac{1}{\lambda^2} \langle L_s L_s u, \varphi \rangle $$ (provided that $L_s u$ belongs to the domain of $L_s$, which is rarely the case!), and in general the righthand side is not equal to $\lambda^{2} \langle (\Delta)^{2s} u, \varphi \rangle$.

$\begingroup$ Ok, I see. Then my real question is the following one: without this trick, how can we prove that $$U(t,x) = \sum_{k=1}^\infty \varphi_k(x) \left( \langle u, \varphi_k \rangle \frac{\sin(\sqrt{\lambda_k} t)}{\sqrt{\lambda_k}} \right)$$ (which is the solution of the fractional wave equation in a regular open set $\Omega$ with initial velocity $u$ and zero initial displacement) is smooth in $x \in \Omega$ if $u$ is smooth? $\endgroup$– RikuCommented Nov 6, 2021 at 13:18

$\begingroup$ I would have tried some energy conservation laws, but I never worked with the wave equation really. There are some recent works on this subject, e.g. DOI:10.1515/fca20180067. $\endgroup$ Commented Nov 6, 2021 at 16:49