The example group given has order 333. The automata program in the Monoid Automata Factory, when given this input:

#aaa,ab'b'b'a'b'a'b _RWS := rec ( isRWS := true, ordering := "shortlex", generatorOrder := [a,A,b,B], inverses := [A,a,B,b], equations := [ [aaa,IdWord], [aBBBABA*b,IdWord] ] );

runs for about six hours but confirms this.

Separately, if you run the above input through the kbprog program from kbmag, with the option -t 1000000 (else it runs for days), it finds a confluent rewriting system after about six minutes; taking one of the last additional relations found in that rewriting system (such as "[B^2abAB,b^2Ab^2]" from the .kbprog output file) gives you this GAP input file:

f := FreeGroup("a","b");; g := f / [ f.1f.1f.1,f.1f.2^-1f.2^-1f.2^-1f.1^-1f.2^-1f.1^-1f.2,f.2^-1f.2^-1f.1f.2f.1^-1f.2^-1f.2^-1f.2^-1f.1f.2^-1*f.2^-1 ]; Size(g);

which prints 333.

Thus two completely independent Knuth-Bendix implementations show this is a finite group of order 333.

The example group given has order 333. The automata program in the Monoid Automata Factory, when given this input:

#aaa,ab'b'b'a'b'a'b
_RWS := rec
(
  isRWS := true,
  ordering := "shortlex",
  generatorOrder := [a,A,b,B],
  inverses := [A,a,B,b],
  equations := 
  [
   [a*a*a,IdWord],
   [a*B*B*B*A*B*A*b,IdWord]
  ]
);

runs for about six hours but confirms this.

Separately, if you run the above input through the kbprog program from kbmag, with the option -t 1000000 (else it runs for days), it finds a confluent rewriting system after about six minutes; taking one of the last additional relations found in that rewriting system (such as "[B^2abAB,b^2Ab^2]" from the .kbprog output file) gives you this GAP input file:

f := FreeGroup("a","b");;
g := f / [ f.1*f.1*f.1,f.1*f.2^-1*f.2^-1*f.2^-1*f.1^-1*f.2^-1*f.1^-1*f.2,f.2^-1*f.2^-1*f.1*f.2*f.1^-1*f.2^-1*f.2^-1*f.2^-1*f.1*f.2^-1*f.2^-1 ];
Size(g);

which prints 333.

Thus two completely independent Knuth-Bendix implementations show this is a finite group of order 333.