Disclaimer : I suspect the question I am about to ask is really hard, but I just want to know the status of such questions.

Thanks to Milnor, we know that the $\pi_1$ any compact manifold with nonnegative Ricci curvature has polynomial growth.

I want to know if anything is known about the opposite direction : which manifolds whose $\pi_1$ has polynomial growth admit metric with nonnegative Ricci curvature ?

At least if you allow non compact manifolds, the answer cannot be 'all of them' (because of Whitehead's three manifold), and I feel like there are compact counter examples but I am not able to cook one.