One approach is to work symbolically and solve a system over the rationals. Choose bounds for the degrees of $S,T$ and write them as $\sum a_m x^i y^j$ where each $a_m$ is a fresh variable. Compute $H(S(x,y),T(x,y))$ - it is a polynomial in $x,y$ with coefficients polynomials in $a_i$. Pick the degree $d_h$ of the homogeneous part and make a system where each monomial coefficient $\ne d_h$ is zero. Solve the system over the rationals. While this will work in theory, solving the system might be quite hard -- couldn't solve your example in 20 minutes. Partially optimistic might be the fact that the system is overdetermined.