The question is well beyond of what is currently known about Poincare duality groups and groups of intermediate growth. The only known case is of PD(2) groups, since they are virtually surface groups. The answer is unclear already for PD(3) groups (conjecturally, they are 3-manifold groups, so they should not have intermediate growth). Note that before Perelman, it was unknown if 3-manifold groups could have intermediate growth. On the other side, there are no known examples of finitely presented (or even $FP_2$) groups of intermediate growth. For all what we know, they are never $FP_2$, while $PD(n)$ groups are $FP_n$ by definition.  

Edit: It is an old theorem (due to Avez, 1970) that amenable fundamental group of a closed nonpositively curved  manifold has to be virtually abelian. The same holds under the mere CAT(0) assumption (no smoothness is needed); this is proven by Burger and Schroeder, Math. Annalen, 1987. Groups of intermediate growth are, of course, amenable.