Notice that you are looking at the 2-category of algebras, bimodules and bimodule homomorphism.
Because of this: if you regard a morphism $f : A \to B$ of algebras as an $A$-$B$ bimodule $B_f$ ($B$ equipped with the obvious right $B$-action and with left $A$-action induced by $f$) then the 2-morphisms that you are looking at are bimodule homomorphism $B_g \to B_f$ given on $B$ by left multiplication with $b \in B$ (this trivially respects the right $B$-action and the equation $f(a)b = b g(a)$ is precisely the condition that it also respects the left $A$-action.)
So you are looking for the $A_\infty$-version of (the maximal higher groupoid inside) the 2-category of algebras, bimodules and bimodule homomorphisms.
Now, in
Berger, Moerdijk, Resolution of coloured operads and rectification of homotopy algebras http://arxiv.org/PS_cache/math/pdf/0512/0512576v2.pdf
there is described in section 6 a model-category theoretic construction of a simplicial category whose objects are $A_\infty$-algebras, morphisms are bimodules of $A_\infty$-algebras, 2-morphisms are bimodule homomorphisms, and so on.
This simplicial category you may think of as presenting an $\infty$-category of $A_\infty$-algebras.
(Notice that this applies to $A_\infty$-algebras over any suitable enriching category, say tor $A_\infty$-spaces You are probably thinking of the standard dg-case, enriched over chain complexes, to which it applies in particular.)
I'll write out more details at
Some paragraphs on this you can also find here:
http://ncatlab.org/nlab/show/model+structure+on+algebras+over+an+operad#InfCatOfMods
in a moment.
