This intuition seems to be only loosely right. There are many smooth compact CY threefolds with large fundamental groups. For instance $\mathbb{Z}/3\times \mathbb{Z}/3$, $\mathbb{Z}/8\times \mathbb{Z}/8$, are allowed fundamental groups and I am pretty sure that those do not act freely on $S^{3}$.
More to the point - the Calabi-Yau threefolds that have these fundamental groups are explictlyexplicitly constructed and we have a pretty good idea of the shape of (at least one of) their slag torus fibrations. For instance, in the first case the Calabi-Yau fibers by genus one curves over a rational elliptic surface, and the slag fibration is compatible with the genus one fibration. In the second case the Calabi-Yau fibers by abelian surfaces and again the slag fibration is compatible. So guided by the holomorphic picture you can easily imagine a situation where you group acts freely on the CY, preserves the slag torus fibration, and the induced action on the base of the fibration is not free. The only thing you can conclude really is that the action of the group on any fiber sitting over a fixed point in the base is free. This is possible to arangearrange on a torus by taking action by translations.
So, even if your fundamelntalfundamental group happens to admit some free action on $S^{3}$, this doesn't mean that the action on the base of the slag fibration will be free. AnAnd, in general, I don't expect it to be free.