This is answered in the proof of Theorem 8.5 of this paper. This says that the rank gradient is zero iff $M$ is prime or $\mathbb{RP}^3\#\mathbb{RP}^3$.
Edit: There's a small step missing from the argument. The argument shows that if $G$ is an orientable connected sum 3-manifold group (not $\mathbb{RP}^3\#\mathbb{RP}^3$), then there exists $G' \lhd G$ of finite index so that $G'$ has positive corank gradient, hence positive rank gradient. Since $d(G')-1\leq [G:G'](d(G)-1)$, if $G$ has rank gradient $0$, so would $G'$ a contradiction (one applies this inequality to $G'\cap H \lhd H$). This applies as well to non-orientable 3-manifolds, where one has to consider the $\mathbb{RP}^2$-decomposition, or just posit that there is a prime cover.