The ratio $\sqrt{2}/2$ is optimal. Set $$a_n = \binom{1/4+n-1}{n-1} r^n \ \mbox{for} \ n \geq 1.$$ Let $Y(r)$ and $X(r)$ be the corresponding sums. I claim that $$ \lim_{r \to 1^-} \frac{X(r)}{Y(r)} = \frac{1}{\sqrt{2}}.$$ We first rewrite the sums a bit: $$X = \sum_n \left( a_n \sum_{i+j+k=n} a_i a_j a_k \right) \quad Y = \sum_n \left( \sum_{i+j=n} a_i a_j \right)^2 .$$ Using generating functions $$\sum_n a_n x^n = \frac{rx}{(1-rx)^{1/4}}$$ $$\sum_n \left( \sum_{i+j=n} a_i a_j \right) x^n = \frac{r^2 x^2}{(1-rx)^{2/4}}.$$ $$\sum_n \left( \sum_{i+j+k=n} a_i a_j a_k \right) x^n = \frac{r^3 x^3}{(1-rx)^{3/4}}.$$ Expanding by the binomial formula, and then using asymtotic formulas for binomial coefficients: $$a_n \sim \frac{n^{-3/4}}{\Gamma(1/4)} r^{n}.$$ $$\sum_{i+j=n} a_i a_j \sim \frac{n^{-2/4}}{\Gamma(2/4)} r^{n}.$$ $$\sum_{i+j+k=n} a_i a_j a_k \sim \frac{n^{-1/4}}{\Gamma(3/4)} r^{n}.$$ So $$a_n \sum_{i+j+k=n} a_i a_j a_k \sim \frac{r^{2n}}{\Gamma(1/4) \Gamma(3/4) n}$$ and so $$X \sim \frac{1}{\Gamma(1/4) \Gamma(3/4)} \log \left( \frac{1}{1-r^2} \right).$$ Similarly, $$Y \sim \frac{1}{\Gamma(2/4)^2} \log \left( \frac{1}{1-r^2} \right).$$ We deduce that, as $r \to 1^-$, the ratio $X/Y$ approaches $\tfrac{\Gamma(1/2)^2}{\Gamma(1/4) \Gamma(3/4)}$. Using the reflection formula for the $\Gamma$ function: $$\frac{\Gamma(1/2)^2}{\Gamma(1/4) \Gamma(3/4)} = \frac{\pi/\sin (\pi/2)}{\pi/\sin (\pi/4)} = \frac{1}{\sqrt{2}}.$$ <hr> I imagine the same would work with $a_n = \eta_R(n) n^{-3/4}$ for any nice enough cut off function $\eta$. To see why we want $a_n \approx n^{-3/4}$, notice that the $n$-th term in the sums defining $X$ and $Y$ is a sum of roughly $n^2$ terms of size roughly $a_n^4$, so we want $n^2 a_n^4 \approx 1/n$ in order for the sum to be just on the border of convergence.