Let me elaborate on the comment I made in the case of a virtually free group. Suppose $G$ is a virtually $F_n$ group, $F_n$ F_n \leq G$a finite-index free group of rank$n$. By Stallings' theorem,$F_n$G$ is a graph of finite groups. In particular, there is a tree $\mathcal{T}$ and an action of $G$ on $\mathcal{T}$, such that $\mathcal{T}/G$ is a finite graph. The action of $F_n$ on $\mathcal{T}$ is faithful and free, since the stabilizer of any vertex in the $G$ action on $\mathcal{T}$ is finite. Consider the kernel $K=ker\{ G \to Aut(\mathcal{T})\}$. Then $K$ is finite since it stabilizes any vertex of $\mathcal{T}$. Then we should think of $\Lambda=\mathcal{T}/(G/K)$ as an orbispace, in the sense of Haefliger, where we attach to each cell of $\Lambda$ the stabilizer in $G/K$ of a lift of the cell to $\mathcal{T}$. Also, $F_n\hookrightarrow G/K$, since $F_n \leq G$ acts faithfully on $\mathcal{T}$. The quotient $\mathcal{T}/F_n$ is a finite graph $\Gamma$ with $b_1(\Gamma)=n$ and no vertices of degree 1. There are only finitely many topological types of such graphs, and only finitely many orbispace covers $\Gamma \to \Lambda$. Thus, there are only finitely many possibilities for $G/K$. These groups are maybe the analogues of $\mathbb{Z},D_\infty$ in your question, or of the Bieberbach groups in Mosher's answer in the $\mathbb{Z}^n$ case.
Let me elaborate on the comment I made in the case of a virtually free group. Suppose $G$ is a virtually $F_n$ group, $F_n$ a free group of rank $n$. By Stallings' theorem, $F_n$ is a graph of finite groups. In particular, there is a tree $\mathcal{T}$ and an action of $G$ on $\mathcal{T}$, such that $\mathcal{T}/G$ is a finite graph. Consider the kernel $K=ker\{ G \to Aut(\mathcal{T})\}$. Then we should think of $\Lambda=\mathcal{T}/(G/K)$ as an orbispace, in the sense of Haefliger, where we attach to each cell of $\Lambda$ the stabilizer in $G/K$ of a lift of the cell to $\mathcal{T}$. Also, $F_n\hookrightarrow G/K$, since $F_n \leq G$ acts faithfully on $\mathcal{T}$. The quotient $\mathcal{T}/F_n$ is a finite graph $\Gamma$ with $b_1(\Gamma)=n$ and no vertices of degree 1. There are only finitely many topological types of such graphs, and only finitely many orbispace covers $\Gamma \to \Lambda$. Thus, there are only finitely many possibilities for $G/K$. These groups are maybe the analogues of $\mathbb{Z},D_\infty$ in your question, or of the Bieberbach groups in Mosher's answer in the $\mathbb{Z}^n$ case.