The answer is Yes, but it fails for some slight variants, so let me first discuss an analogue for sheaves of sets. In that case, this is closely related to the discussion of pro-etale fundamental groups in my paper with Bhatt; that fundamental group classifies such $\mathcal F$.
First, I claim that any sheaf $\mathcal F$ that is fpqc locally constant is representable by a scheme $T\to S$ that is separated and etale (and satisfies the valuative criterion of properness). To see this, we use that separated etale maps descend along fpqc covers of the base, see Tag 0APK for descent of ind-quasi-affine morphisms (which separated etale maps are), and clearly the properties of being separated and of being etale (and the valuative criterion of properness) descend. Thus, it suffices to show the claim fpqc locally on $S$, but then $T$ is just a disjoint union of copies of $S$.
Side remark: It turns out that for general $S$, it is slightly tricky to characterize the class of $\mathcal F$ that are fpqc locally constant. By the above, all of them are representable by schemes separated and etale over $S$, satisfying the valuative criterion of properness. If $S$ has locally a finite number of irreducible components, the converse is true, see Lemma 7.3.9 and Remark 7.3.11 here. In general, the following may however happen: There is some $\mathcal F$ that is not fpqc locally trivial, but for which $\mathcal F\sqcup \mathbb Z$ is fpqc locally trivial, see Example 7.3.12 in loc.cit.
Moreover, such $\mathcal F$ need not be etale locally trivial, as in some of examples we discuss there (for say nodal curves of higher genus). But if the fibres are finite, then $\mathcal F$ is etale locally trivial (even finite etale locally), by parametrizing splittings of $\mathcal F$.
Back to the problem at hand, we can for any finitely generated abelian group $M$ consider the sheaf $\mathcal G$ of isomorphisms between $\mathcal F$ and $M$. Fpqc locally, $S$ decomposes into a part where this is empty, and a part where it is a torsor under $\mathrm{Aut}(M)$, where $\mathrm{Aut}(M)$ has the discrete topology (as $M$ is finitely generated). This implies that $\mathcal G$ is fpqc locally constant, and thus is representable by a separated etale $S$-scheme $T$, whose image is open and closed. After pullback to $T$, we get an isomorphism between $\mathcal F$ and $M$. Varying $M$, we see that $\mathcal F$ is etale locally trivial.
On the other hand, if one drops the assumption that $\mathcal F$ is locally a finitely generated abelian group, then the result is no longer true in general.
Another word of warning: It is not true that a general dualizable group scheme is trivial after a finite etale cover. This is only true for normal schemes; it fails for example for the nodal $\mathbb P^1$. Interestingly, it's equivalent to admitting faithful representations, see here.
