FWIW, there is nothing "truly $\infty$" about this question; the same question can be asked for 1-categories and the answer is the same. In that case it fits into two abstract frameworks:
1: doctrinal adjunction. For a 2-monad $T$ on a 2-category $K$ and a pseudo $T$-algebra map $(g,\bar{g}):A\to B$ whose underlying morphism $g:A\to B$ in $K$ has a left adjoint $f:B\to A$, there is a canonical induced structure of an oplax $T$-algebra map on $f$ (the mate of the pseudo $T$-morphism structure on $g$), and the whole adjunction lifts to the 2-category of $T$-algebras (and pseudo morphisms) if and only if this oplax structure is in fact a pseudo structure.
Now there is a 2-monad $T$ on $\mathrm{Cat}^{\mathrm{ob}(I)}$ whose algebras are functors $I\to \mathrm{Cat}$. The pseudo $T$-morphisms are pseudo natural transformations, and the lax/oplax $T$-morphisms are lax/oplax natural transformations. When doctrinal adjunction is unraveled in the case of this 2-monad, it amounts to exactly the condition mentioned by Denis.
2: property-like structure. You mentioned that the space of adjoints to a given morphism is contractible (if nonempty), i.e. that "having an adjoint" appears to be a mere property of a morphism (rather than structure on it). From this perspective it may be surprising that the adjoints don't fit together. In fact, though, having an adjoint is something in between a "property" and "structure" called a property-like structure: a structure that is unique on objects when it exists, but is not necessarily preserved by morphisms.
One of the simplest examples of property-like structure is "having an identity element" for a semigroup: a semigroup can have at most one identity element, but a semigroup homomorphism need not preserve identities. A more well-known example is "having colimits" for a category: they are unique (up to unique isomorphism) when they exist, but not every functor preserves them (even up to isomorphism). The latter is an example of a special kind of property-like structure called lax-idempotent, in that it is automatically preserved laxly by every morphism (in this case, the comparison map $\mathrm{colim} \circ F \to F \circ \mathrm{colim}$). More precisely, a 2-monad $T$ on a 2-category $K$ is lax-idempotent if every $K$-morphism between $T$-algebras has a unique structure of lax $T$-morphism.
Now there is a 2-monad $T$ on the 2-category $\mathrm{Cat}^{\mathbf{2}}$ whose algebras are functors equipped with a left adjoint, and this 2-monad is lax-idempotent: the unique lax $T$-morphism structure on a commutative square is, again, its mate under the adjunctions. Now your given natural transformation is a functor $I\to \mathrm{Cat}^{\mathbf{2}}$, and the fact that it has adjunctions "pointwise" means that this functor lifts "objectwise" to $T$-algebras. Lax-idempotence of $T$ therefore implies that the functor lifts to the 2-category of $T$-algebras and lax $T$-morphisms; hence it lifts to pseudo $T$-morphisms if and only if all these mates are isomorphisms.
As far as know, neither of these abstract contexts has yet been worked out in the $\infty$ case. But someone should!