Let $R$ be a ring (say noetherian of finite Krull dimension, possibly with additional hypotheses) henselian along the ideal $(p)$, and let $\hat{R}$ be the $p$adic completion. Is it true that the étale cohomology of $R[1/p]$ and $\hat{R}[1/p]$ with mod $p$ coefficients coincide? I believe that this should be true (for $K$theoretic reasons), but I was wondering if there is a direct argument.

$\begingroup$ @BenLim. Presumably when the OP writes "$p$", the OP intends the prime integer $p$. Thus, the fraction field $K$ is characteristic $0$, and every field extension of $K$ is separable. $\endgroup$ – Jason Starr Mar 10 '18 at 9:25
TL;DR: Your expectation is right. In fact, there is a third object to compare with $R[1/p]$ and $\hat R[1/p]$, the affinoid rigid space ${\rm Spf}(\hat R)^{\rm rig}$. The cohomology comparison is given by the GabberFujiwara theorem: see Corollary 6.6.4 in [1].
Let us consider the following general setup: let $(A, I)$ be a henselian couple with $A$ noetherian (i.e. $A$ is henselian along $I$), and let $\hat A$ be the $I$adic completion. We set $$X={\rm Spec}\, A, \quad \hat X = {\rm Spec}\, \hat A,$$ $$U= X  V(I), \quad \hat U = \hat X  V(\hat I) \quad \text{ where }\hat I = I\cdot \hat A.$$ Let $\varepsilon\colon \hat U\rightarrow U$ be the canonical map.
The following is a classical theorem of Elkik.
Theorem 1 (Cor. p. 579 in [2]). The restriction functor $$ \varepsilon^*\colon(\text{finite etale covers of }U)\to (\text{finite etale covers of }\hat U) $$ is an equivalence of categories.
Equivalently, for any finite group $G$, the restriction map $H^1(U, G)\to H^1(\hat U, G)$ is bijective, or $\pi_1(\hat U)\to \pi_1(U)$ is an isomorphism.
The GabberFujiwara theorem (see [1], Cor. 6.6.3) is a farreaching generalization of this, treating the higher cohomology of arbitrary torsion etale sheaves. Actually, the theorem compares the cohomology of $U$ to the cohomology of the rigid space $U^{\rm rig} = {\rm Spf}(\hat A)^{\rm rig}$. Whatever this object is, it is clear from the notation that $U^{\rm rig}$ depends only on $(\hat A, \hat I)$, and in particular we get the following as a corollary (see [1], Cor.6.6.4):
Theorem 2 (The formal base change theorem). Let $\mathscr{F}$ be an etale sheaf of sets (resp. torsion groups, resp. torsion abelian groups) on $U$. Then the pullback map $$ \varepsilon^*\colon H^q(U, \mathscr{F})\longrightarrow H^q(\hat U, \varepsilon^* \mathscr{F}) $$ is an isomorphism for $q=0$ (resp. for $q\leq 1$, resp. for $q\geq 0$).
In particular, at least in good cases (finite cohomological dimension, geometrically unibranch) the ArtinMazur etale homotopy types of $U$ and $\hat U$ are equivalent. (Disclaimer: as written, [1] treats only abelian cohomology, but I'm sure the nonabelian statements can be easily deduced from Elkik).
Addendum. A forthcoming book by Abbes (a sequel to [3]) is expected to contain a new proof of the GabberFujiwara theorem, based on Gabber's affine analog of the proper base change theorem.
Curiosity. In fact in the $p$adic situation you describe, both cohomology groups agree with the cohomology of the fundamental group: Theorem 6.7 in here states that for every noetherian $\mathbf{Z}_{(p)}$algebra $A$ such that $(A, pA)$ is a henselian pair, the scheme $X={\rm Spec}\, A[1/p]$ is a $K(\pi, 1)$. For $\mathbf{F}_p$coefficients, this can be deduced (via GabberFujiwara) from earlier work of Scholze (Theorem 4.9 in [4]).
[1] Fujiwara, K.: Theory of tubular neighborhood in étale topology. Duke Math. J. 80 (1), 15–57 (1995).
[2] Renee Elkik, Solutions d’equations a coefficients dans un anneau henselien, Ann. Sci. Ecole Norm. Sup. (4) 6 (1973), 553–603 (1974). MR 0345966 (49 #10692)
[3] Abbes, A.: Éléments de géométrie rigide. Volume I. Progress in Mathematics, vol. 286, Birkhäuser/Springer Basel AG, Basel (2010).
[4] Scholze, P.: $p$adic Hodge theory for rigid analytic varieties. Forum Math. Pi 1, e177 (2013)