The answer is "no" (see below for an example) but it is almost "yes".  If $M$ does not have any real projective plane boundary components then this follows from Theorem 7 of the paper [A survey on Seifert fibre space conjecture][1] by Jean-Philippe Préaux.  

In general, one has to understand the "Seifert spaces modulo $\mathbb{P}$" introduced by Heil and Whitten.  (This is the first time I've had to think about these - life is simpler when we assume orientability!)  See the above survey paper for references. 

Here is the promised example. Consider the three torus $T = \mathbb{R}^3 / \mathbb{Z}^3$.  There is a $\mathbb{Z}_2$ action via the "antipodal map" $\tau$ that acts as negation on all coordinates.  Note that the fixed point set of $\tau$ is $P = \{(0,0,0), (1/2, 0, 0), \ldots, (1/2, 1/2, 1/2)\}$.  Let $T' = T / \tau$.  So $T'$ is not a three-manifold, due to the orbifold points at $P'$, the image of $P$.  If we remove small neighbourhoods of all of the points of $P'$ we obtain a three manifold $T''$ which has fundamental group $\mathbb{Z}^3 \rtimes \mathbb{Z}_2$.  So (sadly), the fundmental group is neither torsion free nor one of the desired groups. 


  [1]: https://arxiv.org/abs/1202.4142