(Remark: I've edited the original answer to match the change in terminology, and I added some new comments.)
This is a theorem of Rosenstein from the paper $\aleph_0$-categoricity of groups. But note that he uses the terminology bounded order rather than finite exponent.
I say that a group is $\aleph_0$-categorical if its complete theory in the language of groups is $\aleph_0$-categorical (i.e., has a unique countable model).
It is not hard to show that an $\aleph_0$-categorical group has finite exponent. This is also Theorem 1 of Rosenstein's paper. Theorem 2 of his paper provides a converse for abelian groups. So we have:
Theorem. (Rosenstein 1971)
An infinite abelian group is $\aleph_0$-categorical if and only if it has finite exponent.
Appendix. The discussion below has inspired me to add some discussion of the proof of theorems, and some other remarks. The Theorem above comes from two results in Rosenstein's paper.
Theorem 1. An $\aleph_0$-categorical group is has finite exponent.
Proof. Rosenstein uses the followong fact, due to due to Ryll-Nardzewski/Enegler/Svenonius independently.
A countably infinite structure $M$ is $\aleph_0$-categorical if and only if for all $m>0$, the action of $\operatorname{Aut}(M)$ on $M^m$ has finitely many orbits.
Now suppose $\operatorname{Th}(G)$ is $\aleph_0$-categorical. We may assume $G$ is countable. Since elements of distinct orders are in distinct orbits of the action of $\operatorname{Aut}(G)$ on $G$, there is a uniform bound on the orders of torsion elements in $G$. Moreover, $G$ has no elements of infinite order since, if $g$ were such then $(g,g^n)$ for varying $n$ would be in distinct orbits in the action on $G^2$.
Theorem 2. An infinite abelian group of finite exponent is $\aleph_0$-categorical.
See Rosenstein's paper. The main tool is a structure theorem for abelian group of finite exponent as direct sums of cyclic groups. This is apparently called Prufer's First Theorem.
Some remarks:
By YCor's comment about $\mathbb{Q}^{(\omega)}$, one cannot prove Theorem 1 by considering only orbits of singletons. ($\mathbb{Q}$ works also.)
It is easier to prove that an $\aleph_0$-categorical torsion group $G$ has finite exponent. Indeed, if not then by compactness/DLS there is a countable model of $\operatorname{Th}(G)$ with an element of infinite order, which cannot be $G$. But if $G$ is not a torsion group, I don't see a quick way to avoid Ryll-Nardzewski or Omitting Types of some kind (although one can substitute other facts that use these results, e.g., in an $\aleph_0$-categorical structure, the algebraic closure of a finite set is finite).
An $\aleph_0$-categorical group need not be abelian. An example is the countably infinite extraspecial $p$-group (see Definition 5.15 here). On the other hand, there are many results in model theory along the lines of "$\aleph_0$-categorical plus some model theoretic property" implies some abelian-like structure. For example, Bauer, Cherlin, and Macintyre showed that an $\aleph_0$-categorical superstable group is abelian-by-finite.
An infinite group of finite exponent need not be $\aleph_0$-categorical. For example take an infinite finitely generated group of finite exponent (like a Tarski monster). Indeed, by the more general result about algebraic closure stated above, if $G$ is $\aleph_0$-categorical then any finite subset of $G$ generates a finite subgroup.