Does the problem $$(\Delta)^s u=\lambda u \text{ in } \mathbb R^N $$ admit a nontrivial solution when $s\in (0, 1)$ and $\lambda>0.$

$\begingroup$ Yes, of course! Take $u(x) = \cos(\lambda^{1/s} x_1)$, where $x = (x_1, x_2, \ldots, x_N)$. $\endgroup$ – Mateusz Kwaśnicki Apr 17 '19 at 12:50

$\begingroup$ Thanks. So if $u\in H^s(R^N)$, the problem does not admit nontrivial solution. Can it admit positive solutions. $\endgroup$ – Sgas Apr 17 '19 at 18:33

$\begingroup$ (1) There's no solution in $H^s$, but this does not follow from my previous comment: if $u$ is a solution, then $\xi^{2s} \hat{u}(\xi) = \lambda \hat{u}(\xi)$ a.e., which means $\hat{u} = 0$ a.e. Note, however, that there are solutions in $L^p$ if $p$ is large enough! $\endgroup$ – Mateusz Kwaśnicki Apr 17 '19 at 18:45

$\begingroup$ (2) There are no positive solutions. Indeed, in this case $u = \lambda I_{2s} u$, where $I_{2s}$ is the Riesz potential operator; if $u$ is nonzero, then $u(x) \geqslant c_1 (1 + x)^{2s  N}$, and – iterating – $u(x) \geqslant c_2 (1 + x)^{2 k s  N}$ for $k = 1, 2, \ldots$, and eventually $u(x) = \infty$ everywhere, a contradiction. $\endgroup$ – Mateusz Kwaśnicki Apr 17 '19 at 18:46

$\begingroup$ In (2), please can you elaborate how you obtain $u(x)\geq c_{1}(1+x)^{2sN}.$ As $I_{2s}(x, y)= C_{N, s} xy^{2sN}.$ Then the convolution theory. $\endgroup$ – Sgas Apr 18 '19 at 9:34
This is too long for a comment, and actually there is a number of questions that appeared in the comments, so let me summarize my points here.
The answer to the original question: is there a solution to $(\Delta)^s u = \lambda u$, is positive: $u(x) = \cos(\lambda^{1/(2s)} x_1)$ is an example.
More generally, any $u$ equal to the Fourier transform of a finite measure supported on the sphere $\xi = \lambda^{1/(2s)}$ is a solution, and in some sense, these are all solutions.
If $u$ is a solution and $u \in H^{2s}$, then $u$ and $(\Delta)^s u$ have welldefined Fourier transforms (in the $L^2$ sense), and $\widehat{(\Delta)^s u}(\xi) = \xi^{2s} \hat{u}(\xi)$. Therefore, $\xi^{2s} \hat{u}(\xi) = \lambda \hat{u}(\xi)$, which implies that $\hat{u} = 0$ a.e. Consequently, $u = 0$ a.e., that is, there is no nontrivial solution.
A similar argument will work as long as the Fourier transform of $(\Delta)^s u$ can be identified with $\xi^{2s} \hat{u}(\xi)$. This is the case when, for example, $u$ is a tempered distribution and $(\Delta)^s u$ is defined in the sense of $\mathscr{S}'$convolution of tempered distributions. Consequently, every distributional solution is necessarily the inverse Fourier transform of a distribution supported on the sphere $\xi = \lambda^{1/(2 s)}$. Note that there are numerous technicalities here, related to the definition of $\mathscr{S}'$convolution, and I did not attempt to verify all details.
If $u$ is assumed to be nonnegative, then again there are no nontrivial solutions, at least as long as we assume that $(1 + x)^{N  2s} u(x)$ is integrable and $u$ is, say, smooth. (More generally, instead of smoothness we may require that $(\Delta)^s u$ is defined in, say, the distributional sense. Indeed: in this case the convolution of $u$ with a bump function will be smooth, and it will satisfy the equation, and thus it will be zero. By approximation, $u$ is zero a.e.)
In this case we can write Dynkin's formula for $u$: $$ u(x) = \int_B G_B(x, y) (\Delta)^s u(y) dy + \int_{B^c} P_B(x, z) u(z) dz , $$ where $B$ is a (large) ball, $G_B$ is the Green's function, and $P_B$ is the Poisson kernel for $(\Delta)^s$ in $B$. In particular, $$ u(x) \geqslant \int_B G_B(x, y) (\Delta)^s u(y) dy , $$ and sending the radius of $B$ to infinity we find that $$ u(x) \geqslant \int I_{2s}(y  x) (\Delta)^s u(y) dy ,$$ where $I_{2s}$ is the Riesz potential kernel. (Here we need $(\Delta)^s u(y) \ge 0$ in order to apply monotone convergence theorem. In fact, equality holds, but we will not need that.)
Recall that $(\Delta)^s u = \lambda u$, and suppose that $u$ is nonnegative and not equal to zero a.e. If $N < 2 s$, then we obtain $u(x) = \infty$ right away (because $I_{2s}(x) = \infty$). Otherwise, we use a "selfimproving" estimate: since $I_{2s}(y  x) = C_{N,s} y  s^{2s  N} \ge C_{N,s} (1 + x)^{2s  N} (1 + y)^{2s  N}$, we find that $$ u(x) \geqslant C_{N,s} \lambda (1 + x)^{2s  N} \int (1 + y)^{2s  N} u(y) dy . \tag{$\star$} $$ Since $u(x)$ is not everywhere zero, we get $u(x) \geqslant c_1 (1 + x)^{2s  N}$ for some $c_1 > 0$. Plugging this bound into $(\star)$ and estimating the integral leads to an improved estimate $u(x) \geqslant c_2 (1 + x)^{4s  N}$ if $N > 4s$, and $u(x) = \infty$ otherwise. In the former case, we iterate further to get $u(x) \geqslant c_3 (1 + x)^{6s  N}$ if $N > 6 s$, and $u(x) = \infty$ otherwise. After a finite number of steps, we find that $u(x) = \infty$, a contradiction.
All properties of $(\Delta)^s$ used in the above argument seem to be quite standard in the potential theory crowd, and some of them date back to Riesz's 1938 article. For more details and reference, you may like to see my survey Fractional Laplace Operator and its Properties in the Handbook of Fractional Calculus with Applications. Vol. 1: Basic Theory, or my earlier article Ten equivalent definitions of the fractional Laplace operator. (I bet there are more convenient references, feel free to edit and add them if you like).

$\begingroup$ Thanks a lot, I got it. I made a typo in the last comment that $\int_{\mathbb R^N}\frac{u}{1+ x^{N+2s}} $ is finite. $\endgroup$ – Sgas Apr 18 '19 at 14:34

$\begingroup$ If $u$ is in $L^{\infty}(\mathbb R^N)$ then from the equation $u$ is in $C^{\infty}(\mathbb R^N).$ So in this case $(\Delta)^s u$ is defined in the sense of distribution. In fact, $u$ is a classical solution to above problem. $\endgroup$ – Sgas May 17 '19 at 6:17

$\begingroup$ @Sgas: Not sure if I understand correctly. The case $N = 2s$ is indeed famous for its logartithmic appearance, but at least in item 3. above logs are not necessary: as the radius grows, the Green function diverges to infinity, and we get a contradiction right away. $\endgroup$ – Mateusz Kwaśnicki Jul 25 '19 at 19:04