$\newcommand{\Z}{\mathbf{Z}}$No.
Take the Baumslag-Solitar group $$G=\mathrm{BS}(1,3)=\langle t,x\mid txt^{-1}=t^3\rangle=\mathbf{Z}\ltimes_3\mathbf{Z}[1/3]$$$$G=\mathrm{BS}(1,3)=\langle t,x\mid txt^{-1}=x^3\rangle=\mathbf{Z}\ltimes_3\mathbf{Z}[1/3]$$
then the lower central series satisfies $G^1=G$, $G^i=2^{i-1}\mathbf{Z}[1/3]$ for all $i\ge 2$, which has index $2^{i-1}$ in $\mathbf{Z}[1/3]$. So $G^1/G^2\simeq\Z$ and $G^i/G^{i+1}\simeq\Z/2\Z$ for all $i\ge 2$. For $i\ge 2$ define $x_i$ as the nontrivial element of $G^i/G^{i+1}$, which has degree $2$ in the Magnus Lie algebra. Then in the Magnus Lie algebra all $x_i$ commute, and $[t,x_i]=x_{i+1}$.
This Lie $\Z$-algebra is not finitely presented (even after tensoring by $\Z/2\Z$). Indeed for every odd $i\ge 5$, we can define a central extension of this Lie algebra with a central element $z_i$ of degree $i$, with nontrivial additional brackets $[x_j,x_k]=z_i$ if $j+k=i$ (this was essentially defined by Vergne a while ago). So $H_2$ is infinite-dimensional, since it's nonzero in each odd degree $\ge 5$.
(I don't have, for the moment, an example for which it would remain infinitely presented after tensoring with $\mathbf{Q}$.)