No, the smoothness is really needed !
To construct a counter-example, let us choose $X=\mathbb{A}^2$. Consider the action of $\mathbb{Z}/2\mathbb{Z}$ by $(x,y)\mapsto (-x,-y)$.
The quotient of $X$ by this action is the quadric cone $Y\subset \mathbb{A}^3$ defined by $tv=u^2$, where the quotient map $f:X\to Y$ is given by $(x,y)\mapsto (t,u,v)=(x^2,xy,y^2)$.
Now $X$ is Cohen-Macaulay, because it is regular, $Y$ is normal by Serre's criterion, and the fibers of $f$ all have dimension $0$ ($f$ is finite). However, $f$ is not flat. Indeed, if it were, since it is proper, all of its fibers would have the same length. But the fibers of $f$ have length $2$, except the fiber over $(0,0,0)$ that has length $3$.