I think Mark Sapir's cautionary joke that magma is just returning a value of $sin(\pi)$ is actually surprisingly accurate. The "Order" function on a finitely presented group, denoted by magma as GrpFP, Magma returns positive integer if the group is computed to be finite, "Infinity" if the group is known to be infinite (e.g. a map to $\mathbb{Z}$ exists), and "0" when its certificates of infinite order cannot be established, coset enumeration exhausts memory, and magma can't determine the order of the group.

http://magma.maths.usyd.edu.au/magma/handbook/text/773#8529

Having said all that, the magma computation I ran gave a different answer for $G$. Magma is saying $G$ is trivial. I used a machine at the University of Texas, which might have more available memory for coset enumeration than the online magma calculator.

$ > G<a,b>:=Group<a,b|a^\wedge 2,b^\wedge 3,(a*b)^\wedge 7,(a*b*a^\wedge -1*b^\wedge -1)^\wedge 10,((a*b*a^\wedge -1*b^\wedge -1)^\wedge 4*b)^\wedge 7\ >;\\ > Order(G);\\ 1\\ $

However, the orders of $H$ and $I$ don't seem to be computable with the available memory, so they might be infinite and they might not be.

I think Mark Sapir's cautionary joke that magma is just returning a value of $sin(\pi)$ is actually surprisingly accurate. The "Order" function on a finitely presented group, denoted by magma as GrpFP, Magma returns positive integer if the group is computed to be finite, "Infinity" if the group is known to be infinite (e.g. a map to $\mathbb{Z}$ exists), and "0" when its certificates of infinite order cannot be established, coset enumeration exhausts memory, and magma can't determine the order of the group.

http://magma.maths.usyd.edu.au/magma/handbook/text/773#8529

EDIT: The OP uses $[a,b]=a\cdot b^{-1} \cdot a^{-1} \cdot b$, my explanation below is $[a,b]=a\cdot b \cdot a^{-1} \cdot b^{-1}$. The computation of Order with the OP's definition returns a 0, as noted in the comments below.

Having said all that, the magma computation I ran gave a different answer for $G$. Magma is saying $G$ is trivial. I used a machine at the University of Texas, which might have more available memory for coset enumeration than the online magma calculator.

$ > G<a,b>:=Group<a,b|a^\wedge 2,b^\wedge 3,(a*b)^\wedge 7,(a*b*a^\wedge -1*b^\wedge -1)^\wedge 10,((a*b*a^\wedge -1*b^\wedge -1)^\wedge 4*b)^\wedge 7\ >;\\ > Order(G);\\ 1\\ $

However, the orders of $H$ and $I$ don't seem to be computable with the available memory, so they might be infinite and they might not be.

I think Mark Sapir's cautionary joke that magma is just returning a value of $sin(\pi)$ is actually surprisingly accurate. The "Order" function on a finitely presented group, denoted by magma as GrpFP, Magma returns positive integer if the group is computed to be finite, "Infinity" if the group is known to be infinite (e.g. a map to $\mathbb{Z}$ exists), and "0" when its certificates of infinite order cannot be established, coset enumeration exhausts memory, and magma can't determine the order of the group.

http://magma.maths.usyd.edu.au/magma/handbook/text/773#8529

The magma computation I ran gave a different answer for G. Magma is saying $G$ is trivial. I used a machine at the University of Texas, which might have more available memory for coset enumeration than the online magma calculator.

$ > G<a,b>:=Group<a,b|a^\wedge 2,b^\wedge 3,(a*b)^\wedge 7,(a*b*a^\wedge -1*b^\wedge -1)^\wedge 10,((a*b*a^\wedge -1*b^\wedge -1)^\wedge 4*b)^\wedge 7\ >;\\ > Order(G);\\ 1\\ $

However, the orders of $H$ and $I$ don't seem to be computable with the available memory, so they might be infinite and they might not be.

I think Mark Sapir's cautionary joke that magma is just returning a value of $sin(\pi)$ is actually surprisingly accurate. The "Order" function on a finitely presented group, denoted by magma as GrpFP, Magma returns positive integer if the group is computed to be finite, "Infinity" if the group is known to be infinite (e.g. a map to $\mathbb{Z}$ exists), and "0" when its certificates of infinite order cannot be established, coset enumeration exhausts memory, and magma can't determine the order of the group.

http://magma.maths.usyd.edu.au/magma/handbook/text/773#8529

Having said all that, the magma computation I ran gave a different answer for $G$. Magma is saying $G$ is trivial. I used a machine at the University of Texas, which might have more available memory for coset enumeration than the online magma calculator.

$ > G<a,b>:=Group<a,b|a^\wedge 2,b^\wedge 3,(a*b)^\wedge 7,(a*b*a^\wedge -1*b^\wedge -1)^\wedge 10,((a*b*a^\wedge -1*b^\wedge -1)^\wedge 4*b)^\wedge 7\ >;\\ > Order(G);\\ 1\\ $

However, the orders of $H$ and $I$ don't seem to be computable with the available memory, so they might be infinite and they might not be.