The example group given has order 333.  The automata program in the Monoid Automata Factory, when given this input:

<pre>
#aaa,ab'b'b'a'b'a'b
_RWS := rec
(
  isRWS := true,
  ordering := "shortlex",
  generatorOrder := [a,A,b,B],
  inverses := [A,a,B,b],
  equations := 
  [
   [a*a*a,IdWord],
   [a*B*B*B*A*B*A*b,IdWord]
  ]
);
</pre>

runs for about six hours but confirms this.

Separately, if you run the above input through the kbprog program from kbmag, with the option -t 1000000 (else it runs for days), it finds a confluent rewriting system after about six minutes; taking one of the last additional relations found in that rewriting system (such as "[B^2*a*b*A*B,b^2*A*b^2]" from the .kbprog output file) gives you this GAP input file:

<pre>
f := FreeGroup("a","b");;
g := f / [ f.1*f.1*f.1,f.1*f.2^-1*f.2^-1*f.2^-1*f.1^-1*f.2^-1*f.1^-1*f.2,f.2^-1*f.2^-1*f.1*f.2*f.1^-1*f.2^-1*f.2^-1*f.2^-1*f.1*f.2^-1*f.2^-1 ];
Size(g);
</pre>

which prints 333.

Thus two completely independent Knuth-Bendix implementations show this is a finite group of order 333.