I would suggest that you work with the Kummer surface $K$ of $J$ instead of using Mumford coordinates. The advantage is that $K$ is a quartic surface in $\mathbb P^3$; in the case you are considering when the curve has a unique point at infinity, the vanishing of the first coordinate means that the point is in the theta divisor, whereas it is the origin when the first three coordinates vanish (using the standard Kummer coordinates as in the book by Cassels and Flynn). Since your endomorphism commutes with multiplication by $-1$, it induces an endomorphism of $K$. This will be given by a quadruple of homogeneous polynomials of some degree $d$ in the four coordinates; it should not be too hard to figure out what they are from the generic representation in terms of the Mumford representation. Then your problem comes down to checking whether the first of these polynomials vanishes, and if so, whether the next two also vanish. (This assumes that all four polynomials do not vanish simultaneously at some point on $K$.)

When $\gamma$ is multiplication by 2, for example, the polynomials are of degree 4 and can be obtained via

KummerSurface(J)`Delta;

in Magma.

I would suggest that you work with the Kummer surface $K$ of $J$ instead of using Mumford coordinates. The advantage is that $K$ is a quartic surface in $\mathbb P^3$; in the case you are considering when the curve has a unique point at infinity, the vanishing of the first coordinate means that the point is in the theta divisor, whereas it is the origin when the first three coordinates vanish (using the standard Kummer coordinates as in the book by Cassels and Flynn). Since your endomorphism commutes with multiplication by $-1$, it induces an endomorphism of $K$. This will be given by a quadruple of homogeneous polynomials of some degree $d$ in the four coordinates; it should not be too hard to figure out what they are from the generic representation in terms of the Mumford representation. Then your problem comes down to checking whether the first of these polynomials vanishes, and if so, whether the next two also vanish. (This assumes that all four polynomials do not vanish simultaneously at some point on $K$.)

When $\gamma$ is multiplication by 2, for example, the polynomials are of degree 4 and can be obtained via

KummerSurface(J)`Delta;

in Magma.

Added later:

For the curve $$C \colon y^2 = x^5 + 10\,,$$ one choice of polynomials giving multiplication by $\sqrt{5}$ on the Kummer surface is $$\begin{array}{r@{\,}c{\,}l} P_1(x_1,x_2,x_3,x_4) &=& 8000 x_1^3 x_2^2 + 400 x_1^2 x_2 x_4^2 + 200 x_1^2 x_3^2 x_4 + 400 x_1 x_2^2 x_3 x_4 - 600 x_1 x_2 x_3^3 + 5 x_1 x_4^4 + 200 x_2^3 x_3^2 + 10 x_2 x_3 x_4^3 + 10 x_3^3 x_4^2 \\ P_2(x_1,x_2,x_3,x_4) &=& 8000 x_1^4 x_4 + 8000 x_1^3 x_2 x_3 + 400 x_1 x_2^2 x_4^2 + 200 x_1 x_2 x_3^2 x_4 + 400 x_1 x_3^4 - 400 x_2^3 x_3 x_4 + 200 x_2^2 x_3^3 - 5 x_2 x_4^4 + 10 x_3^2 x_4^3 \\ P_3(x_1,x_2,x_3,x_4) &=& 40 x_1^3 x_2 x_4 + 8000 x_1^3 x_3^2 - 8040 x_1^2 x_2^2 x_3 - 200 x_1^2 x_4^3 - 7960 x_1 x_2^4 - 996 x_1 x_2 x_3 x_4^2 + 200 x_1 x_3^3 x_4 + 399 x_2^3 x_4^2 - 598 x_2^2 x_3^2 x_4 - x_2 x_3^4 + 5 x_3 x_4^4 \\ P_4(x_1,x_2,x_3,x_4) &=& 64000 x_1^5 + 2720 x_1^3 x_3 x_4 + 8000 x_1^2 x_2^2 x_4 + 5280 x_1^2 x_2 x_3^2 + 2720 x_1 x_2^3 x_3 - 200 x_1 x_2 x_4^3 - 1528 x_1 x_3^2 x_4^2 + 1600 x_2^5 - 68 x_2^2 x_3 x_4^2 - 64 x_2 x_3^3 x_4 + 52 x_3^5 + x_4^5 \end{array} $$ (they are not unique, since we can add multiples of the defining equation of the Kummer surface).

They were obtained by interpolating data from a number of points. (Magma) Code is available from me on request.