For your question 1), I'd say that probably the best reference is anything on fractional ideals.
For question 2): I don't think this is true.
Let $m,r\in \mathbb N$, $m\geq r\geq 3$ and $A=k[t]$.
Let $B:=k[t^m,t^{m+r},t^{m+r+1},\dots,t^{2m-1}]\subset B:=k[t^m,t^{m+r},t^{m+r+1},t^{m+r+2},\dots]\subset A$ and $B\subset B':=k[t^2,t^3]\subset A$. Let $\mathfrak p=At\cap B$ and $\mathfrak p'=At\cap B'$. Finally let $R=B_{\mathfrak p}$ and $R'=B'_{\mathfrak p'}$
Obviously $\overline R=k[t]_{(t)}$.
It is easy to see that $I=(R:R')=\overline Rt^{m+r}\cap R=(t^{m+r},t^{m+r+1},\dots)$, but this is actually an $\overline R$ ideal, so $(R:I)=\overline R$.
As for conditions on when your condition holds, I don't see a clear one. I can tell you certain patterns that makes it clear how these can fail for subrings of $k[t]$, but those seem a little ad hoc.

