I received the following very explicit answer via private communication: the algebra
$$ A = \mathbb{F}_p[x_1,\ldots,x_6]/(x_1^p,\cdots, x_6^p, x_1x_2 + x_3x_4 + x_5x_6) $$
does not lift to a finite flat $\mathbb{Z}_p$ algebra. (I am still working out the details of why this does not lift.) This is exampe 3.2(4) of Berthelot-Ogus.
