You will have to make an arbitrary choice for the direction of morphisms: is the left adjoint "forward" or "backward"? To prevent that, you can add involutions. The resulting category $\mathbf{InvAdj}$ of involutive categories and adjunctions, that I'll define below, has a lot of interesting structure. It is a dagger category, and in fact the `mother of all dagger categories', as it universally embeds any dagger category. In particular, the full subcategory of (ortho)posets and Galois connections has dagger kernels, dagger biproducts, an an opclassifier. See these two papers. Now for the definition (from 3.1.8 of my thesis):
A functor $* \colon \mathbf{C}^\mathrm{op} \to \mathbf{C}$ is called involutive when $* \circ * = \mathrm{Id}$. Define a category $\mathbf{InvAdj}$ as follows. Objects are pairs $(\mathbf{C},*)$ of a category with an involution. A morphism $(\mathbf{C},*) \to (\mathbf{D},*)$ is functor $F\colon \mathbf{C}^\mathrm{op} \to \mathbf{D}$ that has a left adjoint, where two such functors are identified when they are naturally isomorphic. The identity morphism on $(\mathbf{C},*)$ is the functor $*\colon \mathbf{C}^\mathrm{op} \to \mathbf{C}$; its left adjoint is $*^\mathrm{op} \colon \mathbf{C} \to \mathbf{C}^\mathrm{op}$. The composition of $F\colon \mathbf{C}^\mathrm{op} \to \mathbf{D}$ and $G\colon \mathbf{D}^\mathrm{op} \to \mathbf{E}$ is defined by $G \circ *^\mathrm{op} \circ F \colon \mathbf{C}^\mathrm{op} \to \mathbf{E}$; its left adjoint is $F' \circ * \circ G'$, where $F' \dashv F$ and $G' \dashv G$.
(It is not arbitrary to require a left adjoint instead of a right one. A contravariant functor from $\mathbf{C}$ to $\mathbf{D}$ can be written both as a (covariant) functor $F\colon \mathbf{C}^\mathrm{op} \to \mathbf{D}$ or as a (covariant) functor $F^\mathrm{op}\colon \mathbf{C} \to \mathbf{D}^\mathrm{op}$. The latter version has a right adjoint precisely when the former version has a left adjoint.)