I think it might be more sensible to ask questions like this on the GAP forum, but I can help you with the first of your examples. I haven't tried the second. A standard method of trying to prove finitely presented groups infinite is to look for subgroups of low index and compute their abelian invariants, hoping that you will find a subgroup with an infinite abelian quotient.
To look for subgroups of low index, there is the LowIndexSubgroups command, which finds all subgroups up to a given index, and also GQuotients, which looks for homomorphisms onto specific groups. In your first example, the first approach found me subgroups up to index $12$, and the one of index $12$ turned out to be the second derived group. I then tried systematically looking for homomorhisms of $G''$ onto simple groups. I found homomorphisms onto ${\rm PSL}(2,13)$ and ${\rm PSL}(2,25)$. The kernel of the first of these has a large (and probably infinite) $2$-quotient, but no infinite abelian quotient. I was luckier with the second. Here are the actual GAP commands that prove infiniteness.
gap> F := FreeGroup(3);;
gap> P:=F.1;; Q:=F.2;; R:=F.3;;
gap> p:=3;; q:=4;; r:=4;; a:=5;; b:=1;;
gap> rels:=[P^p,Q^q,R^r,(P*Q)^2,(Q*R)^2,(P*Q*R)^2,
> (Q*R^(r/2 + 1))^a * (R*Q*R^(r/2))^b ];;
gap> G := F/rels;
<fp group on the generators [ f1, f2, f3 ]>
gap> D := CommutatorSubgroup(G,G);;
gap> DD := CommutatorSubgroup(D,D);;
gap> Index(G,DD);
12
gap> homs := GQuotients(DD,PSL(2,25));;
gap> K := Kernel(homs[1]);;
gap> AbelianInvariants(K);
[ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0 ]