I think the answer is yes. Let $A$ be a f.g. group. Assume that $\text{Out}(A)=1$ ($\text{Out}(A)$ finite would probably be enough) and that $A$ contains a central copy of $C^{(\infty)}$, the direct sum of countably many copies of the cyclic group $C$ of prime order $p$. Then $\text{Out}(A\times C)$ is infinite: indeed if $f$ is a homomorphism from $C$ to the center of $A$, then if we set $u_f(a,t)=(af(t),t)$, then $f\mapsto u_f$ injects $\text{Hom}(C,Z(A))$ into $\text{Out}(A\times C)$. Actually all these automorphisms are identity on the finite index subgroup $A$, and this is essentially the reason I don't expect Mark's argument to work.

It remains to find an example of such a group $A$. I expect some complicated nilpotent-by-abelian group over $\mathbf{F}_p[t]$ to work (or Kac-Moody groups??), but I'm not prone to go into computations and maybe somebody else has one example.