I would suggest that you work with the Kummer surface $K$ of $J$ instead of using Mumford coordinates. The advantage is that $K$ is a quartic surface in $\mathbb P^3$; in the case you are considering when the curve has a unique point at infinity, the vanishing of the first coordinate means that the point is in the theta divisor, whereas it is the origin when the first three coordinates vanish (using the standard Kummer coordinates as in the book by Cassels and Flynn). Since your endomorphism commutes with multiplication by $-1$, it induces an endomorphism of $K$. This will be given by a quadruple of homogeneous polynomials of some degree $d$ in the four coordinates; it should not be too hard to figure out what they are from the generic representation in terms of the Mumford representation. Then your problem comes down to checking whether the first of these polynomials vanishes, and if so, whether the next two also vanish. (This assumes that all four polynomials do not vanish simultaneously at some point on $K$.)
When $\gamma$ is multiplication by 2, for example, the polynomials are of degree 4 and can be obtained via
KummerSurface(J)`Delta;
in Magma.