Here is a collection of what I have so far thanks to the answers by [Guntram][1] and [S. Carnahan][2]. Let's denote by $P(n,m)$ the property that $|A^m| \le\binom{n+m-1}{m}$ for all subsets $|A|=n$. 

We have that the only nonabelian $P(2,2)$ groups are of the form $Q_8\times G$ where $G$ is an elementary abelian 2-group, and that $P(3,2)$ groups have to be abelian by Freiman's paper ["On two- and three-element subsets of groups"][3]. 

In ["A characterization of abelian groups"][4], Brailovsky proves that large enough $P(n,2)$ are abelian by showing that $P(n,2)\implies P(n',2)$ for all $n\geq n'\geq 2$, so that the result follows from the previous paragraph.

In ["Small squaring and cubing properties for finite groups"][5], Berkovich, Freiman and Praeger prove that the only nonabelian group with $P(2,3)$ is $S_3$.

On the other hand there are nonabelian groups with $P(n,m)$ whenever $\binom{n+m-1}{m}\geq 2^{2n+1}$ as in [Guntram's answer][1].


  [1]: https://mathoverflow.net/a/77903
  [2]: https://mathoverflow.net/a/78005
  [3]: https://doi.org/10.1007/BF02190175 "Aeq. Math. 22, 140–152 (1981). zbMATH review at https://zbmath.org/0489.20020"
  [4]: https://www.jstor.org/stable/2159119 "Proc. Am. Math. Soc. 117, No. 3, 627–629 (1993), doi:10.2307/2159119. zbMATH review at https://zbmath.org/0789.20022"
  [5]: https://doi.org/10.1017/S0004972700029932 "Bull. Aust. Math. Soc. 44, No. 3, 429–450 (1991). zbMATH review at https://zbmath.org/0728.20020"