Your example is a bit of a red herring, as this is relatively easy for hyperelliptic curves.  A hyperelliptic curve can be reconstructed uniquely from the data of the branch divisor of the degree $2$ map to $\mathbb{P}^1$.  Furthermore, isomorphisms of hyperelliptic curves commute with the degree $2$ map to $\mathbb{P}^1$.  Thus for two hyperelliptic curves, the only issue is whether or not the branch divisors are projectively equivalent, and this is quite straightforward to check.