The ratio $\sqrt{2}/2$ is optimal. Set
$$a_n = \binom{n-7/4}{n-1} r^n \ \mbox{for} \ n \geq 1.$$
Let $Y(r)$ and $X(r)$ be the corresponding sums.
I claim that
$$ \lim_{r \to 1^-} \frac{X(r)}{Y(r)} = \frac{1}{\sqrt{2}}.$$

We first rewrite the sums a bit:
$$X = \sum_n \left( a_n \sum_{i+j+k=n} a_i a_j a_k \right) \quad Y = \sum_n \left( \sum_{i+j=n} a_i a_j \right)^2 .$$

Using generating functions
$$\sum_n a_n x^n = \frac{rx}{(1-rx)^{1/4}}$$
$$\sum_n \left( \sum_{i+j=n} a_i a_j \right) x^n = \frac{r^2 x^2}{(1-rx)^{2/4}}.$$
$$\sum_n \left( \sum_{i+j+k=n} a_i a_j a_k \right) x^n = \frac{r^3 x^3}{(1-rx)^{3/4}}.$$

Expanding by the binomial formula, and then using asymtotic formulas for binomial coefficients:
$$a_n \sim \frac{n^{-3/4}}{\Gamma(1/4)} r^{n}.$$
$$\sum_{i+j=n} a_i a_j \sim \frac{n^{-2/4}}{\Gamma(2/4)} r^{n}.$$
$$\sum_{i+j+k=n} a_i a_j a_k \sim \frac{n^{-1/4}}{\Gamma(3/4)} r^{n}.$$

So
$$a_n \sum_{i+j+k=n} a_i a_j a_k \sim \frac{r^{2n}}{\Gamma(1/4) \Gamma(3/4) n}$$
and so
$$X \sim \frac{1}{\Gamma(1/4) \Gamma(3/4)} \log \left( \frac{1}{1-r^2} \right).$$
Similarly,
$$Y \sim \frac{1}{\Gamma(2/4)^2} \log \left( \frac{1}{1-r^2} \right).$$

We deduce that, as $r \to 1^-$, the ratio $X/Y$ approaches $\tfrac{\Gamma(1/2)^2}{\Gamma(1/4) \Gamma(3/4)}$. Using the reflection formula for the $\Gamma$ function:
$$\frac{\Gamma(1/2)^2}{\Gamma(1/4) \Gamma(3/4)} = \frac{\pi/\sin (\pi/2)}{\pi/\sin (\pi/4)} = \frac{1}{\sqrt{2}}.$$

I imagine the same would work with $a_n = \eta_R(n) n^{-3/4}$ for any nice enough cut off function $\eta$. To see why we want $a_n \approx n^{-3/4}$, notice that the $n$-th term in the sums defining $X$ and $Y$ is a sum of roughly $n^2$ terms of size roughly $a_n^4$, so we want $n^2 a_n^4 \approx 1/n$ in order for the sum to be just on the border of convergence.

For example, the same thing happens if we take $a_n = n^{-c}$ and let $c \to 3/4^{+}$. Approximating sums by integrals gives
$$\sum_{\begin{matrix} i+j=n \\ i,j > 0 \end{matrix}} i^{-c} j^{-c} = n^{1-2c} \sum_{\begin{matrix} i+j=n \\ i,j > 0 \end{matrix}} \frac{1}{n} \left( \frac{i}{n} \right)^{-c} \left( \frac{j}{n} \right)^{-c} \sim n^{1-2c} \int_{\begin{matrix} x+y=1 \\ x,y > 0 \end{matrix}} x^{-c} y^{-c} dx = n^{1-2c} \frac{\Gamma(1-c)^2}{\Gamma(2-2c)}$$
$$\mbox{and}\ \sum_{\begin{matrix} i+j+k=n \\ i,j,k > 0 \end{matrix}} i^{-c} j^{-c} k^{-c} = n^{2-3c} \sum_{\begin{matrix} i+j+k=n \\ i,j,k > 0 \end{matrix}} n^{-2} \left( \frac{i}{n} \right)^{-c} \left( \frac{j}{n} \right) \left( \frac{k}{n} \right)^{-c}$$
$$\sim n^{2-3c} \int_{\begin{matrix} x+y+z=1 \\ x,y,z > 0 \end{matrix}} x^{-c} y^{-c} z^{-c} dx dy = n^{2-3c} \frac{\Gamma(1-c)^3}{\Gamma(3-3c)}.$$
So, for $c>3/4$, we have
$$X \sim \sum n^{2-4c} \frac{\Gamma(1-c)^3}{\Gamma(3-3c)} = \zeta(4c-2) \frac{\Gamma(1-c)^3}{\Gamma(3-3c)}$$
$$\mbox{and similarly} \ Y \sim \zeta(4c-2) \frac{\Gamma(1-c)^4}{\Gamma(2-2c)^2}.$$
So
$$\lim_{c \to 3/4^+} \frac{X}{Y} = \frac{\Gamma(1/2)^2}{\Gamma(1/4) \Gamma(3/4)}=\frac{1}{\sqrt{2}}.$$