Assume the group is not free. Then its Cayley graph has a nontrivial loop. Applying shifts of the group you get a loop which starts at any element.

If the group grows faster then polynomially, then the number of disjoint cycles and therefore first Betti number grows faster than polynomial.

Assume the group is not free. Then its Cayley graph has a nontrivial loop. Applying shifts of the group you get a loop which starts at any element.

If the group grows faster then polynomially, then the number of disjoint cycles and therefore first Betti number grows faster than polynomial.

So, the answer is "yes" --- only free groups and groups of polynomial growth.