Let me consider the case of the Fermat quartic. Since you are talking about Castelnuovo's criterion, I guess you are interested in the projective situation.

The $1$-form $$\psi \colon =\frac{dx}{y^3}=-\frac{dy}{x^3}$$ is holomorphic on the projective curve $F$, so the $2$-form $$p^* \psi \wedge q^* \psi = p^* \frac{dx}{y^3} \wedge q^* \frac{dx}{y^3}$$ (where $p, \, q$ are the natural projections) is holomorphic on $F \times F$.

One easily checks that $p^* \psi \wedge q^* \psi$ is invariant by the subgroup of $G \subset \operatorname{Aut}(F \times F)$ generated by the three involutions $$i_x \colon (x,\, y) \mapsto (-x, \, y), \quad i_y \colon (x,\, y) \mapsto (x, \, -y), \quad \iota \colon (x,\, y) \mapsto (y, \, x),$$ hence it descends to a non-zero holomorphic $2$-form on $S=(F \times F)/G$.

Thus $h^0(S, \, K_S) \geq 1$, in particular $S$ is not rational.

Let me consider the case of the Fermat quartic. Since you are talking about Castelnuovo's criterion, I guess you are interested in the projective situation.

The $1$-form $$\psi \colon =\frac{dx}{y^3}=-\frac{dy}{x^3}$$ is holomorphic on the projective curve $F$, so the $2$-form $$\tau:=p^* \psi \wedge q^* \psi = p^* \frac{dx_1}{y_1^3} \wedge q^* \frac{dx_2}{y_2^3}$$ (where $(x_1, \, y_1)$ are the coordinates on the first factor, $(x_2, \, y_2)$ those on the second factor and $p, \, q$ are the natural projections) is holomorphic on $F \times F$.

One easily checks that $\tau$ is invariant by the two involutions $i_1, \, i_2$ of $F \times F$ defined by $$i_1:=i_x \times i_x \colon ((x_1,\, y_1), \, (x_2, \, y_2)) \mapsto ((-x_1,\, y_1), \, (-x_2, \, y_2)),$$ $$i_2:=i_y \times i_y \colon ((x_1,\, y_1), \, (x_2, \, y_2)) \mapsto ((x_1,\, -y_1), \, (x_2, \, -y_2)).$$ Moreover, denoting by $\iota$ the involution switching the two factors, namely $$\iota \colon ((x_1,\, y_1), \, (x_2, \, y_2)) \mapsto ((x_2,\, y_2), \, (x_1, \, y_1)),$$ we have $\iota^* \tau = - \tau$.

Thus, the tensor $\tau \otimes \tau$ is invariant by the action of the subgroup $G \subset \operatorname{Aut}(F \times F)$ given by $G = \langle i_1, \, i_2, \, \iota\rangle$, hence it descends to a non-zero, global holomorphic tensor on $S=(F \times F)/G$.

This shows that $h^0(S, \, K_S^{\otimes 2}) \geq 1$, in particular $S$ is not rational.