There a simple way to make this work:
Say T:V->X is a map of inner-product vector spaces. You can view V as a category, where Hom(v,w) is a singleton set containing one real number, the inner product <v,w>, and similarly for X.
Composition, a binary operation, is defined (stupidly, as in any category with singleton hom-sets) as follows:
Comp_{uvw} : Hom(u,v)xHom(v,w) -> Hom(u,w)
by (<u,v>,<v,w>) |-> <u,w>
Then the adjoint T*:X->V satisfies
<Tv,x>=<v,T*x>, i.e. Hom(Tv,x)=Hom(v,T*x)
, meaning it is a right adjoint to T (in a very strong sense: we have equality of these hom-sets instead of just natural isomorphism).
The triviality of this example reflects the fact that that T and T* are called "adjoint" simply because they belong on opposite sides of a comma :)
In general, if H is any function of two variables, we can say that g is right adjoint to f "with respect to H" if H(f(a),b)=H(a,g(b)), and say that "adjoint functors" are "adjoint with respect to Hom" (up to natural isomorphism, of course).