Regarding extending existing techniques: One common method of proving residual finiteness of hyperbolic groups is cubulation. For example, one-relator groups with torsion and small cancellation groups were shown to be residually finite by proving that a finite index subgroup acts on a CAT(0) cubical complex in a specific (special...) way. However, cubulated hyperbolic groups are always linear, and M. Kapovich proved that there are non-linear hyperbolic groups. Therefore, cubulation cannot be extended to all hyperbolic groups.

@MarkGrant Thanks for pointing this out. I've fixed it now, and also tried to made it all slightly clearer.

@MarkGrant Yes, $\phi^{-1}(d)$ works. The point is that $G$ is shown to be free-by-$D_{\infty}$, so the pre-image of any $\mathbb{Z}$ subgroup of $D_{\infty}$ will work.

Are you assuming free-by-cyclic means {finitely generated free}-by-cyclic? I know this is usual, but a preprint of Kielak and Linton dropped last week which is very relevant to the more general setting.

@Mike The standard example of a Baumslag-Solitar group with non-finitely generated (outer) automorphism group is $BS(2, 4)$, due to Collins and Levine in the 1980s ("Automorphisms and Hopficity of certain Baumslag-Solitar groups" Arch. Math. (1983)); Levitt gave a geometric reason for this non-finite generation in a 2007 G&T paper (link). Theorem 5.2 of this gives a positive answer to your question in the "nice" case of (Generalised) Baumslag-Solitar groups with no $BS(1, n)$ subgroups, $n>1$.

@HJRW Sorry, yes, parameterised by $G$. Then each $\phi_x$ for $x\in N$ is trivial, so this gives a map $Q\to\operatorname{Aut}(H)\to\operatorname{Out}(H)$. These maps can then be shown to be injective here (the map to $\operatorname{Aut}$ is pretty clearly injective, by for example Britton's Lemma, while the map to $\operatorname{Out}$ is also injective as none of these $\phi_x$ are inner because, as $G$ has trivial centre and $N\neq G$, no inner automorphism of $H$ fixes every $g\in G$).

@HJRW Finitely presentability can be salvaged from Moishe Kohan's answer, but this likely sacrifices finite dimensionality: Take the HNN-extension $H=\langle G, t\mid t^{-1}xt=x,\:x\in N\rangle$. Then the automorphisms $\phi_x: g\mapsto g, t\mapsto x^{-1}tx$ for $x\in N$ gives an embedding of $Q$ into $\operatorname{Out}(H)$.

@BenjaminSteinberg I don't disagree - I focussed too much on the title of the tread. But I did make the reaction when I learned this result!

Sorry, wasn't quite thinking correctly - the Guba/Albert-Lawrence result means that the nice descriptions of solutions sets, via Makanin-Razborov diagrams and as EDT0L languages, hold for all systems of equations, rather than just finite systems.

There is a remarkable fact underlying this work, which is possibly worthy of its own answer: Every system of equations over a free monoid/free group having a finite number of variables is equivalent to a finite subsystem of it. This was the Ehrenfeucht conjecture, proven by Albert-Lawrence and Guva in the 1980s.