This is a reference question:
Let $\psi(x)$ be the psi-Chebyshev function. Is there any unconditional result in the literature that proves that there exists $0<a<2$ such that $$ \int_2^x (\psi(y)-y)^2 \mathrm dy =O(x^{a}) ?$$
This is a reference question:
Let $\psi(x)$ be the psi-Chebyshev function. Is there any unconditional result in the literature that proves that there exists $0<a<2$ such that $$ \int_2^x (\psi(y)-y)^2 \mathrm dy =O(x^{a}) ?$$
Impossible as this would imply that $\frac{\zeta'(s)}{\zeta(s)}+\frac1{s-1}$ is analytic on $\Re(s)\ge 1/2$.
Your bound $$\int_1^x |\psi(y)-y|^2 dy = O(x^a)$$ for some $a < 2$ implies (with Cauchy-Schwarz inequality) that $$\int_x^{2x} (\psi(y)-y)dy = O(x^{a/2+1/2}) \implies\int_1^2 (\psi(xt)-xt)dt = O(x^{a/2-1/2}) $$ i.e. $$s\int_1^\infty\int_1^2 (\psi(tx)-tx)dt x^{-s-1}dx= \left(\frac{-\zeta'(s)}{\zeta(s)}-\frac{s}{s-1}\right)\int_1^2 t^s dt-f(s)$$
extends analytically from $\Re(s) > 1$ to $\Re(s) > a/2-1/2$ which is absurd.
(where $f(s)=s\int_1^2 t^s\int_1^t x^{-s}dxdt$ is an irrelevant entire term)
In Theorem 1 of
Brent, Richard P.; Platt, David J.; Trudgian, Timothy S., The mean square of the error term in the prime number theorem, ZBL07569752.
it is shown that for sufficiently large $x$ one has the lower bound $$ \int_x^{2x} |\psi(y)-y|^2\ dy \geq 0.000186 x^2$$ and on RH one has a matching upper bound $$ \int_x^{2x} |\psi(y)-y|^2\ dy \leq 0.8603 x^2$$ Thus your desired bound cannot hold for any $a<2$. (If RH fails, then this integral is known to grow strictly faster than $x^2$.)
In that paper several previous results of this type are also discussed. Interestingly, they show that even on RH, the quantity $\frac{1}{x^2} \int_0^x |\psi(y)-y|^2\ dy$ cannot converge to a limiting value as $x \to \infty$, even though it is bounded both above and below (there are terms that oscillate like $x^{i\gamma}$ for some gap $\gamma$ between zeroes).