Abdelmalek Abdesselam's suggestion that you try Groebner bases is reasonable for the degre-6 examples that mention in your update to the original question.

Let me spell out the steps explicitly to check whether two forms homogeneous f and g of the same degree in [x,y] are equivalent:

Let $a,b,c,d$ be the matrix-entry coordinates on ${\rm GL}_2$ and consider $f(ax+by,cx+dy)$.

Calculate $f(ax+by,cx+dy)-g(x,y)$ and extract the list $J$ of coefficients of $x^i y^j$. This list represents an ideal in $k[a,b,c,d,t]/(t(ad-bc)-1)$, the coordinate ring of ${\rm GL}_2$. The ideal corresponds to the subscheme of ${\rm GL}_2$ transporting $f$ into $g$, i.e. it gives the conditions on a section of ${\rm GL}_2$ for that section to transform $f$ into $g$.

Compute a Groebner basis of $K = J+[t(ad-bc)-1]$ in $k[a,b,c,d,t]$. If $K$ is not equivalent to $[1]$---if it is not the unit ideal---then there is a section of the transporter over $\overline{k}$ (Nullstellensatz). Computing such a section tells one how to transform $f$ into $g$. If $K$ is equivalent to $[1]$---if it is the unit ideal---then ${\rm GL}_2$ does not transform $f$ into $g$ over $\overline{k}$.

If you would prefer to work with a subgroup scheme $G$ of ${\rm GL}_2$, then add the generators of the ideal defining $G$ to $K$ before computing the Groebner basis.

I tried a pair of your forms, checking that $h(x,y)$ is equivalent to $f(x,y)$. Maple returned the following Groebner basis (lexicographic order t>a>b>c>d) instantly:
$$
[d^{12}+2118775924690448809984*d^6+93687714211574708957308664016389973143977984, 5*d^7+14472313805968991556993024*c+96131361823291542077440*d, d^7+20557263928933226643456*b+19226272364658308415488*d, -5*d+704*a, 1039*d^{10}-13614472272996909891715072*d^4+1144692921788849487441900451832894830569062400*t]
$$

This Groebner basis tells us to do the following to find the invertible matrix transforming h into f:

a. Solve the following degree-12 equation for $d$:
$$
d^{12}+2118775924690448809984d^6+93687714211574708957308664016389973143977984 = 0.
$$
Any root is fine.

b. Use the following equation to find $c$ from $d$:
$$
5d^7+96131361823291542077440d +14472313805968991556993024c= 0.
$$

c. Use the following equation to find $b$ from $d$:
$$
d^7+19226272364658308415488d +20557263928933226643456b= 0.
$$

d. Use the following equation to find $a$ from $d$:
$$
-5d+704a=0.
$$

The transporter scheme is defined over $k$ (which is $\mathbf{Q}$ in this case), and you can use it to study the fields over which $f$ and $h$ are equivalent, if you wish.

I tried the same procedure for transforming $g(x,y)$ into $f(x,y)$ and found that one could construct a matrix for the transformation as follows:

a. Solve the equation
$$
d^6-121740744925904896 = 0
$$
to find $d$.

b. Set $c=0$ and $b=0$.

c. Use the equation
$$
-5d+704a = 0
$$
to get $a$.

There is another type of transformation matrix with $a=0$ and $d=0$, whose description I omit. (The parameterizing scheme also has degree 6.)

In both cases, we get a degree-12 scheme for the total transporter, since your forms have automorphism group schemes of degree 12. (The transporter, when nonempty, is a bitorsor under the automorphism group schemes of source and target forms.)

This straightforward procedure could become unusable if you had a large number of forms to check or if you wanted to work with forms of higher degree. For studying a few forms of low degree, however, it is both speedy and easy---and it does not require looking up any invariants.

9more comments