Ah, great, I see now, thanks! It seems like there really should not be any higher $L_\infty$-operations then. I'm not sure how to check this (i.e. check that the resulting $L_\infty$-algebra quasi-isomorphism type of $\mathrm{CC}^*(A)$ is an $A_\infty$-algebra quasi-isomorphism invariant of $A$), since functoriality of Hochschild cochains under $A_\infty$-algebra quasi-isomorphisms is a bit messy, so I will leave the question open in case anybody else has suggestions for this.

Thanks @StefanWaldmann - is $V$ an $A_\infty$-module over $A$ in your comment? What I really want to know is whether the differential and bracket (with vanishing higher operations) give the correct $L_\infty$-structure on $\mathrm{CC}^*(A)$. E.g. if you replaced $A$ with a quasi-equivalent dg-algebra, would this result in a quasi-equivalent $L_\infty$-structure on $\mathrm{CC}^*$?