The ratio $\sqrt{2}/2$ is optimal. Define $a_n(r,c)$ by the power series
$$\sum_{n=1}^{\infty} a_n(r,c) x^n = \frac{x}{(1-rx)^c}.$$
Let $Y(r,c)$ and $X(r,c)$ be the corresponding sums.
I claim that
$$\lim_{c \to 1/4^+} \lim_{r \to 1^-} \frac{X(r,c)}{Y(r,c)} = \frac{1}{\sqrt{2}}.$$

We first rewrite the sums a bit:
$$X = \sum_n \left( a_n \sum_{i+j+k=n} a_i a_j a_k \right) \quad Y = \sum_n \left( \sum_{i+j=n} a_i a_j \right)^2 .$$

Using generating functions
$$\sum_n \left( \sum_{i+j=n} a_i a_j \right) x^n = \frac{x^2}{(1-rx)^{2c}}.$$
$$\sum_n \left( \sum_{i+j+k=n} a_i a_j a_k \right) x^n = \frac{x^3}{(1-rx)^{3c}}.$$

Expanding by the binomial formula, and then using asymtotic formulas for binomial coefficients:
$$a_n \sim \frac{n^{c-1}}{\Gamma(c)} r^{n-1}.$$
$$\sum_{i+j=n} a_i a_j \sim \frac{n^{2c-1}}{\Gamma(2c)} r^{n-2}.$$ 
$$\sum_{i+j+k=n} a_i a_j a_k \sim \frac{n^{3c-1}}{\Gamma(3c)} r^{n-3}.$$ 

So $X$ is a sum of terms like $n^{4c-2}$, as is $Y$. As long as $c>1/4$, we deduce
$$X \sim \frac{\Gamma(4c-1)}{\Gamma(c) \Gamma(4c) (1-r)^{4c-1}} \ \mbox{as}\ r \to 1^-$$
$$Y \sim \frac{\Gamma(4c-1)}{\Gamma(2c)^2 (1-r)^{4c-1}} \ \mbox{as}\ r \to 1^-.$$
(If $c<1/4$, then the singularity $1/(1-r)^{4c-1}$ doesn't blow up as $r \to 1^-$, so we have to actually compute the sums.) We see that
$$\lim_{r \to 1^-} \frac{X(r,c)}{Y(r,c)} = \frac{\Gamma(2c)^2}{\Gamma(c) \Gamma(3c)}.$$
Sending $c \to 1/4^{+}$ and using the reflection formula for the $\Gamma$ function we obtain
$$\frac{\Gamma(1/2)^2}{\Gamma(1/4) \Gamma(3/4)} = \frac{\pi/\sin (\pi/2)}{\pi/\sin (\pi/4)} = \frac{1}{\sqrt{2}}.$$

<hr> 

I imagine the same would work with $a_n = \eta_R(n) n^{-3/4}$ for any nice enough cut off function $\eta$. To see why we want $a_n \approx n^{-3/4}$, notice that the $n$-th term in the sums defining $X$ and $Y$ is a sum of roughly $n^2$ terms of size roughly $a_n^4$, so we want $n^2 a_n^4 \approx 1/n$ in order for the sum to be just on the border of convergence.