Your example is a bit of a red herring, as this is relatively easy for hyperelliptic curves. A hyperelliptic curve can be reconstructed uniquely from the data of the branch divisor of the degree $2$ map to $\mathbb{P}^1$. Furthermore, isomorphisms of hyperelliptic curves commute with the degree $2$ map to $\mathbb{P}^1$. Thus for two hyperelliptic curves, the only issue is whether or not the branch divisors are projectively equivalent, and this is quite straightforward to check.
I believe that more generally this problem is difficult, but I don't claim to be an expert on the topic.