Yes, this is true for any odd exponent $2k-1$ on place of 3.
First of all, $a_m$ is a monic polynomial in $\ell$ of degree $m-1$, and these polynomials are known as Chebyshev polynomials of second kind: if $\ell=2\cos x$, then $a_m=\frac{\sin mx}{\sin x}$. So it suffices to prove that $A:=\prod_{j=1}^n (a_j^{2k-1}+\ldots+a_n^{2k-1})$ is divisible by $B:=a_1\ldots a_n$ as a polynomial (then the ratio is a monic polynomial in $\ell$ with integer coefficients). The roots of $a_m$ are simple and are equal to $2\cos \pi t/m$, $t=1,\ldots,m$. So, if $0<u<v$ are integers and $u,v$ are coprime, the number $2\cos \pi u/v$ is a root of $B$ of multiplicity $\lfloor n/v\rfloor:=q$. Thus it suffices to prove that there exist at least $q$ indexes $j$ for which $a_j^{2k-1}+\ldots+a_n^{2k-1}$ is divisible by $a_v$, call these indexes good. Consider $q$ sets $\Delta_i:=\{iv,iv-1,\ldots,(i-1)v+1\}$, $i=1,\ldots,q$. I claim that each $\Delta_i$ contains a good index $j$. Denote $n=qv+r$, $0\leqslant r<v$. Look at the remainders of $a_i$'s modulo $a_v$. They are $0=a_0,a_1,a_2,\ldots,a_{v-1},0$, then $-a_{v-1},-a_{v-2},\ldots,-a_1,0$, then $a_1,a_2,\ldots$ etc. Thus $a_{qv+j}\equiv -a_{qv-j} \pmod {a_v}$ for all $j=0,1,\ldots$ that implies that $qv-r\in \Delta_q$ is good, so is $n-2v\in \Delta_{q-1}$, $qv-r-2v\in \Delta_{q-2}$ etc.