This can be proved using the strategy of Fedor Petrov and a theorem from the following paper:
Haselgrove, C. B., Some theorems in the analytic theory of numbers, J. Lond. Math. Soc. 26, 273-277 (1951). ZBL0043.04704.
Let $63/64 < \theta < 1$.
According to Theorem A of Hasselgrove, if $m$ is a sufficiently large odd number, then $m$ is the sum of three primes $p_1$, $p_2$, $p_3$ with $|p_i-m/3| < m^{\theta}$. Putting $m = 2 \ell-3$, we obtain
$$4 \ell = 2(p_1+1) + 2(p_2+1) + 2(p_3+1).$$
The Paley construction gives a Hadamard matrix of size $2(p+1)$ for any odd prime $p$. Since $p_i = (1/3) m + O(m^{\theta})$, we have $2(p_i+1) = (4/3) \ell + O(\ell^{\theta})$.
The exponent $63/64$ has been improved on by many other authors; for example, Matomakai, Maynard and Shao push it down to $11/20$. But the OP just asked for each of the matrix sizes to be comparable to $\ell$, so I'll stop here.