It seems more natural to define the category to consist of those finitely generated $R$-modules whose generic fiber is 1-dimensional (without choosing a preferred basis).  I use this viewpoint below.

If $R$ is Gorenstein then you can take $X=R$ since the dualizing complex for a Gorenstein local ring is concentrated in a single degree and as such as an invertible module (so the Hom above is the same as an RHom). This conceptually explains your second case with $R \cap K_0 = O_{K_0}$ for a subfield $K_0$ over which $K$ is quadratic since in such cases $R$ is Gorenstein (as after completion over $O_{K_0}$ we are confronted with a "quadratic order" over a discrete valuation ring, which is always monogenic and hence easily checked to be Gorenstein). 

In general $R$ is Cohen-Macaulay, so it is tempting to try to take $X$ to be its dualizing module (as the dualizing complex for a CM local ring is supported in a single degree), but if not projective (equivalently, $R$ not Gorenstein) then presumably the discrepancy between Hom and RHom may create some problems.  No doubt the experts in commutative algebra can supply a counterexample or explain why it is a non-issue in these circumstances. 

In fact, one is led to wonder (in the absence of any motivation being given for the question) whether the setup is simply "wrong": it is *always* true by taking $X$ to be the dualizing module (put in degree 0) that on the derived category $D^b_c(R)$ of "bounded complexes of $R$-modules with finitely generated homologies" that $T \mapsto {\rm{RHom}}(T,X)$ is an involutory auto-equivalence and ${\rm{RHom}}(X,X) = R$. That is, the dualizing module *always* works if you work in the appropriate derived category setting (which eliminates the 1-dimensionality restriction on the generic fiber, etc.). In the Gorenstein case we can apply ${\rm{H}}^0$'s throughout to recover the more concrete assertion with ordinary Hom's in that case. Is that not adequate for whatever motivated the question?