Yes, for any henselian local (e.g., complete local noetherian) ring $A$ with residue field $\kappa$ and any connected reductive $\kappa$-group $G$, there exists a connected reductive $A$-group scheme $\mathbf{G}$ with special fiber $G$. Note that your hypothesis on the $\mathbf{F}_p$-group just says that this $\mathbf{F}_p$-group is connected reductive (there is no need to state a condition separately on its geometric fiber over $k$).

To prove this, it seems simplest to use the crutch of the existence of Chevalley groups over $\mathbf{Z}$. More specifically, there does exist a split reductive $A$-group $\mathbf{H}$ such that $H := \mathbf{H}_{\kappa}$ is a Galois-twisted form of $G$. Over any field at all, connected reductive groups always admit a (unique up to isomorphism) quasi-split inner form. Let $G'$ denote the quasi-split inner form of $G$, so $G'$ is a quasi-split form of the split $\kappa$-group $H$.

We will first solve the lifting problem for $G'$, and then solve it for $G$. Put another way, we will first solve it when $G$ is quasi-split, and then will have to address the behavior under passage to inner forms.
Handling the latter step will be more serious, involving automorphism schemes of reductive groups over rings, so it is worth noting that when $\kappa$ is finite or separably closed then inner forms are always trivial and hence $G$ is always quasi-split. Thus, *for such special $\kappa$ the more technically difficult task of going beyond the quasi-split case does not arise*.

How does one make quasi-split forms of a split connected reductive group? These are classified (up to isomorphism) by degree-1 Galois cohomology valued in the automorphism group of the associated based root datum $(R, \Delta)$. In other words, the pointed set ${\rm{H}}^1(\kappa, {\rm{Aut}}(R, \Delta))$ of conjugacy classes of continuous maps ${\rm{Gal}}(\kappa_s/\kappa) \rightarrow {\rm{Aut}}(R, \Delta)$ classifies all such possibilities. In concrete terms, the Galois-twisting of $H$ to create the quasi-split $G$ is given through "pinned automorphisms". But the notion of "pinned automorphism" makes sense for any pinned split reductive group over any base, and in particular we can choose a pinning of $\mathbf{H}$ over $A$ to make ${\rm{Aut}}(R, \Delta)$ act on $\mathbf{H}$.

Thus, if we fix a finite Galois extension $\kappa'/\kappa$ so that $G$ is made from $H$ through twisting by a homomorphism
$c: {\rm{Gal}}(\kappa'/\kappa) \rightarrow {\rm{Aut}}(R, \Delta)$
and view $c$ as a function on ${\rm{Aut}}(A'/A)$ for the (unique) local finite etale $A$-algebra $A'$ with residue field $\kappa'/\kappa$ then we can carry out the same "Galois-twisting" directly over $A'$ (using "Galois descent" relative to the finite etale cover ${\rm{Spec}}(A') \rightarrow {\rm{Spec}}(A)$) to make an $A$-form $\mathbf{G}$ of $\mathbf{H}$ with special fiber $G$. **We are now done if $\kappa$ is finite or separably closed**.

For the general case, we now have a reductive $A$-group scheme $\mathbf{G}'$ whose reduction $G'$ is an inner form of $G$, and we seek to make a reductive $A$-group $\mathbf{G}$ with reduction $G$; the split $\mathbf{H}$ and $H$ have done their work above and will no longer be used.

By the very definition of inner forms, $G$ is obtained from $G'$ through twisting against a cohomology class in ${\rm{H}}^1(\kappa, {G'}^{\rm{ad}})$ where ${G'}^{\rm{ad}} := G'/Z_{G'}$ is the adjoint semisimple central quotient of $G'$. The natural map
$$f:{\rm{H}}^1(\kappa, {G'}^{\rm{ad}}) \rightarrow {\rm{H}}^1(\kappa, {\rm{Aut}}_{G'/\kappa})$$
into the pointed set of isomorphism classes of forms of $G'$ is generally not injective. Rather, by Proposition 39(ii) in section 5.5 of Chapter I of Serre's book "Galois cohomology" applied to the short exact sequence of $\kappa_s$-point arising from the short exact sequence of smooth $\kappa$-groups
$$1 \rightarrow {G'}^{\rm{ad}} \rightarrow {\rm{Aut}}_{G'/\kappa} \rightarrow E_{G'} \rightarrow 1$$
(with $E_{G'}$ the etale component group of the automorphism scheme of $G'$), there is a natural action by $E_{G'}(\kappa)$ on ${\rm{H}}^1(\kappa, {G'}^{\rm{ad}})$ and the $f$-fibers are the $E_{G'}(\kappa)$-orbits. By using more work with the etale topology over rings in place of Galois cohomology over fields, we can define the separated etale $A$-group $E_{\mathbf{G}'}$ similarly using the automorphism scheme ${\rm{Aut}}_{\mathbf{G}'/A}$. This reduces our task to showing that the pointed set of torsors ${\rm{H}}^1(A, {\mathbf{G}'}^{\rm{ad}})$ maps surjectively to ${\rm{H}}^1(\kappa, {G'}^{\rm{ad}})$ and that $E_{\mathbf{G}'}(A) \rightarrow E_{G'}(\kappa)$ is surjective too.

Since $A$ is henselian, for *any* smooth $A$-scheme $X$ the natural map $X(A) \rightarrow X(k)$ is surjective due to the Zariski-local structure of smooth maps and the various ways of characterizing henselian local rings; see [EGA, IV$_4$, 18.5.17]. Thus, setting $X = E_{\mathbf{G}'}$ settles our second surjectivity task. It remains to prove surjectivity of
${\rm{H}}^1(A, {\mathbf{G}'}^{\rm{ad}}) \rightarrow {\rm{H}}^1(\kappa, {G'}^{\rm{ad}})$. For the smooth affine $A$-group $\mathscr{G} := {\mathbf{G}'}^{\rm{ad}}$, the vector bundle $\mathfrak{g} := {\rm{Lie}}(\mathscr{G})$ over $A$ is a finite free $A$-module (since $A$ is local) and the adjoint representation
${\rm{Ad}}_{\mathscr{G}}: \mathscr{G} \rightarrow
{\rm{GL}}(\mathfrak{g}) = {\rm{GL}}_N$
has trivial kernel.

[The difficult Theorem 5.3.5 in the linked article on reductive group schemes in the question posed shows that this monomorphism is a closed immersion. Likewise, further arguments show that the associated coset sheaf for the adjoint representation of any adjoint semisimple group over any scheme is represented by an affine scheme, necessarily smooth over the base -- the key point is to use effective descent for affineness to work etale-locally on the base to reduce to working with split groups and thereby reduce to working over $\mathbf{Z}$ that is Dedekind, to which Theorem C.2.5 in the PhD thesis of Brandon Levin is applicable. We will not need either of these refinements.]

Now comes the trick to turn our H$^1$-surjectivity problem into a more tangible "geometric" lifting problem that makes contact with the henselian property of $A$. Consider the $\mathscr{G}$-torsor
$${\rm{GL}}(\mathfrak{g}) \rightarrow Z := {\rm{GL}}(\mathfrak{g})/\mathscr{G}$$
where $Z$ is a sheaf for the fppf (or etale) topology over $A$. This defines a natural map $Z(A) \rightarrow {\rm{H}}^1(A, {\mathbf{G}'}^{\rm{ad}})$
compatible with the natural map $Z(\kappa) \rightarrow {\rm{H}}^1(\kappa, {G'}^{\rm{ad}})$.

The vanishing of ${\rm{H}}^1(\kappa, {\rm{GL}}_N)$ implies that the map $Z(\kappa) \rightarrow {\rm{H}}^1(\kappa, {G'}^{\rm{ad}})$ is surjective (!), so our surjectivity problem for ${\rm{H}}^1$'s reduces to proving the surjectivity of the natural map $Z(A) \rightarrow Z(\kappa)$.
If $Z$ were known to be a smooth $A$-scheme then we would be done again by [EGA, IV$_4$, 18.5.17] since $A$ is henselian. But $Z$ seems to be "just" a quotient sheaf, without evident geometric substance, so what can we do?

We saw above that $Z$ is in fact a smooth affine $A$-scheme, but this isn't needed: it would suffice to know that $Z$ is a quasi-separated algebraic space (then necessarily smooth over $A$), as that provides enough "geometric structure" on $Z$ to apply the same EGA result with the aid of the henselian local ring at a given $\kappa$-point of $Z$ (which makes sense for quasi-separated algebraic spaces). The definition of $Z$ as an fppf quotient sheaf associated to the equivalence relation defined by the *free* action arising from translation by an inclusion between finite type flat $A$-group schemes implies that $Z$ is an algebraic space by a deep theorem of Artin concerning quotients by fppf equivalence relations for schemes. This thereby settles the case of general residue fields $\kappa$.