That is not an answer. I want to give an example where the argument of Erdős does not work directly.
Consider an action of group $\Gamma$ on $\mathbb R^3$ generated by the reflections in two lines $x=z=0$ and $x+1=z=0$ and a glide rotation $(x,y,z)\mapsto (-y,x,z+1)$.
The group $\Gamma$ has 2 (up to conjugation) maximal finite subgroups. (So $2\leqslant 2^3$ — the conjecture is fine.) Both of these groups isomorphic to $\mathbb Z_2$. The factor $X=\mathbb R^3/\Gamma$ is a manifold with two singular circles, say $\Sigma_1$ and $\Sigma_2$.
Let us try to mimic argument of Erdős. Take subsets $X_i$ of $X$ of midpoints $m$ between $x\in X$ and a closest $x_0\in\Sigma_i$ to $x$. As in the argument of Erdős we have* $\mathrm{vol}\\, X_i>\tfrac{1}{2^3}\cdot\mathrm{vol}\\, X$. BUT $X_1\cap X_2$ has interior points and here argument brakes into parts.
Comment.
(*) Since fixed point sets are 1-dimensional, it would be enough to take $m\in [xx_0]$ such that $\tfrac{|mx_0|}{|xx_0|}=\tfrac1{2\sqrt{2}}$. But even in this case one has interior points in $X_1\cap X_2$ (the borderline in this example seems to be $\tfrac13$).