Since $H$ is reduced and $Y$ is smooth over $H$ (I am assuming that everything is finite type over $k$, so smooth and formally smooth are the same) we see that $Y$ is reduced.

So the problem is the following: show that if $Y \subset R \times \mathbb P^n$ is open and reduced, and the projection $Y \to R$ is surjective (taking into account the remark to this effect in the comments), then $R$ is reduced.

Here is the proof:
Let $x$ be a point of $R$, and let $y$ be a point of $Y$ lying over $x$. Recalling
that $\mathbb P^n$ is the union of $n + 1$ open subsets isomorphic to
$\mathbb A^n$, we may assume
that $y \in R\times \mathbb A^n$ (for an appropriate choice of one of these $n+1$ copies).
The stalk $\mathcal O_{Y,y}$ is then equal to a localization of $\mathcal O_{R,x}[x_1,\ldots,x_n]$. It is reduced by assumption, and so $\mathcal O_{R,x}$ is reduced.
Since $x \in R$ was arbitrary, we see that $R$ is reduced.