The answer is yes, fully-faithful functors are stable under co-base change.
This is a model independent statement and so we can in particular take $\infty$-category to mean Segal categories. Then this follows directly from Cor. 16.6.2 in the arXiv version of Carlos Simpson's book "Homotopy theory of higher categories", in particular see the proof of this corollary.
More precisely, the full-faithfulness condition in terms of hom spaces also makes sense for Segal precategories, and it is clearly preserved for (homotopy) pushouts of these. Simpson's Cor. 16.6.2 shows that this condition is still preserved after you re-complete to get a Segal category again.
Simpson proves this very generally in the context of M-enriched Segal precategories with very mild conditions on the model category M. By using Simpson's result and varying M you get the analogous statement in many other contexts, not only for $\infty$-categories. For example you also get this in the canonical homotopy theory of ordinary categories, and also in the homotopy theory of Cat-enriched Segal categories. Since this later is equivalent to the homotopy theory of bicategories and in this case the fully-faithfulness can be expressed in a homotopically independent way, you can deduce this for bicategories as well.