Edit: the following does not answer Ben's question. It gives an example of the subring fixed by $G_m$ being not CM, while the question asked about the subscheme of fixed points, see the comments for more details.
Let $R$ be the (homogenous) cone of a curve $C$ of genus $g>0$, for example $R=\mathbb C[x,y,z]/(x^3+y^3+z^3)$. Let $S=R[u,v]$, $X=\text{Spec}(S)$ and $G_m$ acts by
$a.(x,y,z,u,v) = (ax,ay,az,a^{-1}u, a^{-1}v)$.
Then $A= S^{G_m}$ would be a homogenous coordinate ring for $Y= C\times \mathbb P^1$, so it is not Cohen-Macaulay (if $A$ is CM, it would mean that $H^1(Y,\mathcal O_Y)=0$, impossible, see herehere for an explanation).
(I learned this idea from Hochster, let me try to find a reference)