If your CY manifold is simply connected, the base of the torus-fibration will have to be simply connected too, since a homotopically non-trivial loop downstairs would lift to a loop upstairs which does not bound a disc. In 3 dimensions, that's the end of the story by the Poincaré conjecture.
I'll try to explain via homological mirror symmetry (HMS) why, even in higher dimensions, the base of the SYZ fibration should be a rational homology-sphere. This only applies to "strict" CY manifolds.
Say we have a special Lagrangian torus-fibration $\check{X}\to B$, and we would like to understand the Fukaya category as the derived category of a mirror $X$, defined over some field $K$ of characteristic zero (depending on the formulation of the Fukaya category, $K$ might be the field of rational or complex Novikov series; optimists think that $\mathbb{C}$ could also be a possibility for $K$).
A basic aspect of HMS is the prediction that the mirror to a smooth torus-fiber $F_b$ will be a skyscraper sheaf $\mathcal{O}_{X,x}$ on $X$. That prediction gives rise to another: that the mirror $L$ to the structure sheaf $\mathcal{O}_X$ should be a Lagrangian section of $\check{X}\to B$. The reason is that $\mathrm{Ext}^\ast(\mathcal{O}_X,\mathcal{O}_{X,x})=H^\ast(\mathcal{O}_{X,x})=K$, so by HMS one should have $HF(L,F_b)=K$ for each fibre $F_b$. Taking Euler characteristics of the latter isomorphism, we get $[L]\cdot [F_b]=1$. So $L$ is at least a homology-section, and we guess that it should be a true section. In particular, $H^\ast(L;K)=H^\ast(B;K)$.
By a "strict" CY $n$-manifold I mean that as well as trivial canonical bundle, one has $H^i(\mathcal{O}_X)=0$ for $0<i<n$. (In the setting of complex manifolds, this means that the holonomy is exactly $SU(n)$.) By Serre duality, $H^n(\mathcal{O}_X)=K$. Hence $\mathrm{Ext}^*(\mathcal{O}_X,\mathcal{O}_X)= H^\ast(\mathcal{O}_X)$ is isomorphic as a graded $K$-algebra to $H^\ast(S^n;K)$. On the other hand, $\mathrm{Ext}^*(\mathcal{O}_X,\mathcal{O}_X)\cong HF(L,L)$ by HMS. One makes the reasonable guess that $HF(L,L)\cong H^\ast(L;K)$, and infers that $H^*(B;K) = H^*(L;K)\cong H^*(S^n;K)$.
Edit: Ah, I think we don't need to guess at the last stage! The DGA of cochains computing $\mathrm{Ext}^*(\mathcal{O}_X,\mathcal{O}_X)$ is formal - over $\mathbb{C}$, we get that from Deligne-Griffiths-Morgan-Sullivan plus Hodge by using a Dolbeault model. By HMS, $CF(L,L)$ is then formal as an $A_\infty$-algebra. The Oh spectral sequence $H^\ast(L;K) \Rightarrow HF(L,L)$ must then surely degenerate at $E_1$, so $H^\ast(L)\cong HF(L,L)$.
