No. Let $R = \mathbb{Z}_p[C_p]$ and consider the $R$-modules $M = R$ and $N = \mathfrak{m}$, the maximal ideal of the local ring $R$. Let $A,B$ the matrices in $GL_p(\mathbb{Z}_p)$ giving the action of a generator of the cyclic group $C_p$ on $M$ and $N$ respectively. Then $M \otimes_{\mathbb{Z}_p} \mathbb{Q}_p \cong \mathbb{Q}_p[C_p] \cong N \otimes_{\mathbb{Z}_p} \mathbb{Q}_p$ as $\mathbb{Q}_p[C_p]$-modules, so $A$ is conjugate to $B$ in $GL_p(\mathbb{Q}_p)$.

Write $R \cong \mathbb{Z}_p[T] / \langle (1 + T)^p - 1\rangle$ and let $f$ be the image of $\frac{1}{T}( (1 + T)^p - 1)$ in $R$; then it is not difficult to see that $N \cong R/\langle T\rangle \oplus R / \langle f(T)\rangle$ as $R$-modules. This means that $N/pN$ is not a cyclic $R/pR$ module. However $M/pM = R/pR$ is a cyclic $R/pR$ module, so $M/pM$ is not isomorphic to $N/pN$ over $\mathbb{F}_p[C_p]$. Hence the reductions of $A$ and $B$ cannot be conjugate in $GL_p(\mathbb{F}_p)$.

No. Let $R = \mathbb{Z}_p[C_p]$ and consider the $R$-modules $M = R$ and $N = \mathfrak{m}$, the maximal ideal of the local ring $R$. Let $A,B$ the matrices in $GL_p(\mathbb{Z}_p)$ giving the action of a generator of the cyclic group $C_p$ on $M$ and $N$ respectively. Then $M \otimes_{\mathbb{Z}_p} \mathbb{Q}_p \cong \mathbb{Q}_p[C_p] \cong N \otimes_{\mathbb{Z}_p} \mathbb{Q}_p$ as $\mathbb{Q}_p[C_p]$-modules, so $A$ is conjugate to $B$ in $GL_p(\mathbb{Q}_p)$.

Write $R \cong \mathbb{Z}_p[T] / \langle (1 + T)^p - 1\rangle$ and let $f$ be the image of $\frac{1}{T}( (1 + T)^p - 1)$ in $R$; then it is not difficult to see that $N \cong R/\langle T\rangle \oplus R / \langle f\rangle$ as $R$-modules. This means that $N/pN$ is not a cyclic $R/pR$ module. However $M/pM = R/pR$ is a cyclic $R/pR$ module, so $M/pM$ is not isomorphic to $N/pN$ over $\mathbb{F}_p[C_p]$. This is because the $C_p$-coinvariant spaces of $M/pM$ and $N/pN$ are $1$ and $2$-dimensional over $\mathbb{F}_p$, respectively. 

Hence the reductions of $A$ and $B$ cannot be conjugate in $GL_p(\mathbb{F}_p)$.