You should check the case of binary sectics since that is the best understood. The $GL_2(\mathbb C)$-invariants are called absolute invariants and denoted by $i_1, i_2, i_3$. Two binary sectics are projectively equivalent if they have the same absolute invariants; see the following for details.
- Invariants of Binary Forms, V. Krishnamoorthy, T. Shaska, H. Voelklein
- Jun-ichi Igusa, Arithmetic variety of moduli for genus two. Ann. of Math. (2) 72 1960 612–649.
For the case of binary octavics such absolute invariants are also explicitly known. There are $t_1, \dots , t_6$ invariants which satisfy some algebraic relation among them. Their definitions can be found in
- T. Shaska, Some remarks on the hyperelliptic moduli of genus 3, Communications in Algebra, Volume 42, Issue 9, September 2014, pages 4110-4130.