No, not even when you restrict to groupoids. In fact there's a counterexample in the book of Fresse that you cite (and it's explicitly said in §I.5.2.2 that the arity-wise inverse don't always assemble to an operad morphism, in fact). Consider the equivalence $\mathsf{PaB} \to \mathsf{CoB}$ from the operad of parenthesized braids to the operad of colored braids (which are both operads in groupoids). It doesn't admit a weak inverse, as in fact there does not even exist any morphism in the reverse direction. This can be seen on the level of objects: there's no way to send the only element of arity 2 in $\mathsf{CoB}$ to an element of $\mathsf{PaB}(2)$ and get an operad morphism (because $m \in \mathsf{CoB}(2)$ satisfies $m \circ_1 m = m \circ_2 m$, i.e. it is associative, but no element of $\mathsf{PaB}(2)$ satisfies this relation).
Indeed, these "categorical equivalences" are equivalences as in "weak equivalences of spaces", you can only invert them under some conditions (eg cofibrant and fibrant objects). The point of these categorical equivalences is that they induce equivalences of simplicial operads after applying the classifying space functor, see the volume II of the book mentioned above.