The answer is no. Let $Y_1(N)$ be the modular curve over $\mathbb{F}_p$ of level $\Gamma_1(N)$ with $N \ge 4$ and coprime to $p$. This is a fine moduli space of elliptic curves with level structure with universal family $E \to Y_1(N)$. Let $k$ be an algebraic closure of the generic point of $Y_1(N)$ and let $A=E_k$. Then $A$ is ordinary hence $A[p^{\infty}] \simeq \mathbb{Q}_p/\mathbb{Z}_p \oplus \mu_{p^{\infty}}$ since $k$ is algebraically closed. It is however not true that $A$ is isomorphic to an elliptic curve $A'$ defined over a finite field. Indeed, by construction the morphism $\operatorname{Spec} k \to Y_1(N)$ does not factor through a closed point.
The reason your question has a negative answer is that there are positive dimensional families of abelian varieties with (geometrically fiberwise) constant $p$-divisible groups. In the case of supersingular abelian varieties, however, such families should not exist.
Assume for simplicity that we are in the polarized case, so that we have access to a moduli space. Let $\mathcal{A}_{g,N}$ be the moduli space of principally polarized abelian varieties with level $N$ structure (here $N$ is coprime to $p$). Let $x:\operatorname{Spec} k \to \mathcal{A}_{g,N}$ be a morphism from an algebraically closed field landing in the supersingular locus $\mathcal{A}_{g,N, \mathrm{ss}}$. Fix a point $y:\operatorname{Spec} \mathbb{F}_{q} \to \mathcal{A}_{g,N, \mathrm{ss}}$ for some finite field $\mathbb{F}_q$, let $(A_y, \lambda_y)$ be the corresponding principally polarized supersingular abelian variety and let $Y=B[p^{\infty}]$.
Let $\operatorname{RZ}_{(Y, \mu)}$ be the Rapoport--Zink space associated to $(Y, \mu)$, which is the functor on $\mathbb{F}_q$-schemes sending a test scheme $T$ to the set of isomorphism classes of pairs $(\mathbb{X}, \beta)$, where $\mathbb{X}$ is a $p$-divisible group over $T$ and where $\beta:Y_{T} \dashrightarrow \mathbb{X}$ is a quasi-isogeny (compatible with the polarizations in a precise way). Then $\operatorname{RZ}_{(Y, \mu)}$ is representable by a formal scheme which is formally locally of finite type over $\mathbb{F}_q$, and there is a Rapoport--Zink uniformization map $\Theta_{y}:\operatorname{RZ}_{(Y, \mu)} \to \mathcal{A}_{g,N, \mathrm{ss}}$ sending $(\mathbb{X}, \beta)$ to the unique principally polarized abelian variety $(B,\mu)$ over $T$ equipped with a quasi-isogeny $\phi:(A_{y,T}, \lambda_{y,T}) \dashrightarrow (B,\mu)$ (compatible with the polarizations in a precise way) such that $B[p^{\infty}]=\mathbb{X}$ and such that $\phi$ induces $\beta$ on the level of $p$-divisible groups. [If $\beta$ is an actual isogeny, then $B=A_{y,T}/\ker \beta$, for the general case see Section 6.13 of Rapoport--Zink].
Rapoport and Zink prove (Theorem 6.24) that $\Theta_{y}$ is formally etale and surjective on the level of $K$-points for all algebraically closed field $K$. In particular, we see that $x$ is in the image of $\Theta_{y}$. It also follows from this surjectivity that $x$ factors through an $\overline{\mathbb{F}}_p$ point of $\mathcal{A}_g$, if and only if $x=\Theta_{y}(z)$ for some $\overline{\mathbb{F}}_p$-point $z$ of $\operatorname{RZ}_{(Y, \mu)}$. Note that the latter statement is equivalent to $(A_x[p^{\infty}], \lambda_x)$ being defined over a finite field, and the former statement is equivalent to $(A_x, \lambda_x)$ being defined over a finite field.