First let's note that by the paper, $Q(R)$ is the total ring of fractions of $R$, i.e. it is the localization of $R$ by the subset of non-zero-divisors. The natural map $R \to Q(R),\,r \mapsto \frac{r}{1}$ is an embedding.
Let $R \le S \le Q(R)$ be commutative rings. Then $R:S \cong Hom_R(S,R)$ as $R$-modules. An isomorphism is given by $\varphi: r \mapsto (s \mapsto rs)$ with inverse $f \mapsto f(1)$.
By commutativity of the rings, $\varphi$ is obviously a hom. of $R$-modules and because of $\varphi(r)(1)=r$, $\varphi$ is injective.
Surjectivity: Let $f: S \to R$ be $R$-linear and let $s\in S$. Write $s=a/b$ with $a,b\in R$ and $b$ a non-zero-divisor. Since $bs \in R$, $bsf(1)=f(bs)=bf(s)$ and since $b$ is not a zero-divisor, we have $f(s)=f(1)s$. This shows (i) $f(1)\cdot S = f(S) \subseteq R$ i.e. $f(1) \in R:S$ and (2) $f = \varphi(f(1))$.