Let me consider the case of the Fermat quartic. Since you are talking about Castelnuovo's criterion, I guess you are interested in the projective situation.  

The $1$-form $$\psi \colon =\frac{dx}{y^3}=-\frac{dy}{x^3}$$ is holomorphic on the projective curve $F$, so the $2$-form $$p^* \psi \wedge q^* \psi = p^* \frac{dx}{y^3} \wedge  q^* \frac{dx}{y^3}$$ (where $p, \, q$ are the natural projections) is holomorphic on $F \times F$.   

One easily checks that $p^* \psi \wedge q^* \psi$ is invariant by the subgroup of $G \subset \operatorname{Aut}(F \times F)$ generated by the three involutions  $$i_x \colon (x,\, y) \mapsto (-x, \, y), \quad i_y \colon (x,\, y) \mapsto (x, \, -y), \quad \iota \colon (x,\, y) \mapsto (y, \, x),$$ hence it descends to a non-zero holomorphic $2$-form on $S=(F \times F)/G$.

Thus $h^0(S, \, K_S) \geq 1$, in particular $S$ is *not* rational.