Let $a_n$ be a sequence of positive numbers and define $$A_n=\sum_{k=1}^{n-1} a_k a_{n-k}.$$ I am interested in the supremum of the following quantity $X/Y$ where $$X=\sum _{i=1}^{\infty } \left(\sum _{j=1}^{\infty } a_i a_{i+j} A_j\right)$$ and $$Y=\sum _{j=1}^{\infty } A_j^2.$$
It have the inequality $X/Y\le \sqrt{2}/{2}$ but maybe this is not sharp.
Experiments with mathematica suggest that $X/Y \le 1/2$.
The ratio $\sqrt{2}/2$ is optimal. Set $$a_n = \binom{n-7/4}{n-1} r^n \ \mbox{for} \ n \geq 1.$$ Let $Y(r)$ and $X(r)$ be the corresponding sums. I claim that $$ \lim_{r \to 1^-} \frac{X(r)}{Y(r)} = \frac{1}{\sqrt{2}}.$$
We first rewrite the sums a bit: $$X = \sum_n \left( a_n \sum_{i+j+k=n} a_i a_j a_k \right) \quad Y = \sum_n \left( \sum_{i+j=n} a_i a_j \right)^2 .$$
Using generating functions $$\sum_n a_n x^n = \frac{rx}{(1-rx)^{1/4}}$$ $$\sum_n \left( \sum_{i+j=n} a_i a_j \right) x^n = \frac{r^2 x^2}{(1-rx)^{2/4}}.$$ $$\sum_n \left( \sum_{i+j+k=n} a_i a_j a_k \right) x^n = \frac{r^3 x^3}{(1-rx)^{3/4}}.$$
Expanding by the binomial formula, and then using asymtotic formulas for binomial coefficients: $$a_n \sim \frac{n^{-3/4}}{\Gamma(1/4)} r^{n}.$$ $$\sum_{i+j=n} a_i a_j \sim \frac{n^{-2/4}}{\Gamma(2/4)} r^{n}.$$ $$\sum_{i+j+k=n} a_i a_j a_k \sim \frac{n^{-1/4}}{\Gamma(3/4)} r^{n}.$$
So $$a_n \sum_{i+j+k=n} a_i a_j a_k \sim \frac{r^{2n}}{\Gamma(1/4) \Gamma(3/4) n}$$ and so $$X \sim \frac{1}{\Gamma(1/4) \Gamma(3/4)} \log \left( \frac{1}{1-r^2} \right).$$ Similarly, $$Y \sim \frac{1}{\Gamma(2/4)^2} \log \left( \frac{1}{1-r^2} \right).$$
We deduce that, as $r \to 1^-$, the ratio $X/Y$ approaches $\tfrac{\Gamma(1/2)^2}{\Gamma(1/4) \Gamma(3/4)}$. Using the reflection formula for the $\Gamma$ function: $$\frac{\Gamma(1/2)^2}{\Gamma(1/4) \Gamma(3/4)} = \frac{\pi/\sin (\pi/2)}{\pi/\sin (\pi/4)} = \frac{1}{\sqrt{2}}.$$
I imagine the same would work with $a_n = \eta_R(n) n^{-3/4}$ for any nice enough cut off function $\eta$. To see why we want $a_n \approx n^{-3/4}$, notice that the $n$-th term in the sums defining $X$ and $Y$ is a sum of roughly $n^2$ terms of size roughly $a_n^4$, so we want $n^2 a_n^4 \approx 1/n$ in order for the sum to be just on the border of convergence.
For example, the same thing happens if we take $a_n = n^{-c}$ and let $c \to 3/4^{+}$. Approximating sums by integrals gives $$\sum_{\begin{matrix} i+j=n \\ i,j > 0 \end{matrix}} i^{-c} j^{-c} = n^{1-2c} \sum_{\begin{matrix} i+j=n \\ i,j > 0 \end{matrix}} \frac{1}{n} \left( \frac{i}{n} \right)^{-c} \left( \frac{j}{n} \right)^{-c} \sim n^{1-2c} \int_{\begin{matrix} x+y=1 \\ x,y > 0 \end{matrix}} x^{-c} y^{-c} dx = n^{1-2c} \frac{\Gamma(1-c)^2}{\Gamma(2-2c)}$$ $$\mbox{and}\ \sum_{\begin{matrix} i+j+k=n \\ i,j,k > 0 \end{matrix}} i^{-c} j^{-c} k^{-c} = n^{2-3c} \sum_{\begin{matrix} i+j+k=n \\ i,j,k > 0 \end{matrix}} n^{-2} \left( \frac{i}{n} \right)^{-c} \left( \frac{j}{n} \right) \left( \frac{k}{n} \right)^{-c}$$ $$\sim n^{2-3c} \int_{\begin{matrix} x+y+z=1 \\ x,y,z > 0 \end{matrix}} x^{-c} y^{-c} z^{-c} dx dy = n^{2-3c} \frac{\Gamma(1-c)^3}{\Gamma(3-3c)}.$$ So, for $c>3/4$, we have $$X \sim \sum n^{2-4c} \frac{\Gamma(1-c)^3}{\Gamma(3-3c)} = \zeta(4c-2) \frac{\Gamma(1-c)^3}{\Gamma(3-3c)}$$ $$\mbox{and similarly} \ Y \sim \zeta(4c-2) \frac{\Gamma(1-c)^4}{\Gamma(2-2c)^2}.$$ So $$\lim_{c \to 3/4^+} \frac{X}{Y} = \frac{\Gamma(1/2)^2}{\Gamma(1/4) \Gamma(3/4)}=\frac{1}{\sqrt{2}}.$$
Maybe I am mistaken, but $a_n=1$ gives $X/Y=3/2$ higher than your claim according to Maple.
Session:
a:=n->1:A:=n->sum(a(k)*a(n-k),k=1..n-1);
X:=sum(sum(a(i)*a(i+j)*A(j),j=1..N),i=1..N);Y:=sum(A(j)^2,j=1..N);
limit(X/Y,N=infinity);
3/2
