$\def\cO{\mathcal{O}}\def\cE{\mathcal{E}}$I will show that the answer to (1) is yes if and only if the vector bundle $f_{\ast} \cO_X$ on $U$ extends to a vector bundle on $S$. Moreover, if $S \setminus U$ is codimension $\geq 2$ in $S$, I will show that the extension is unique as well.

This condition is obviously necessary.

Here is an example of a case where $f_{\ast} \cO_X$ does not extend. Start with Steven Landsburg's example (well, Swan's example, but Landsburg is easier to cite) of a vector bundle $V$ on $U:=\mathbb{A}^3 \setminus \{ 0 \}$ which doesn't extend $S:=\mathbb{A}^3$. If we take global sections of $V$ over $U$, we get $M:= \{ (f,g,h) \in k[x,y,z]^3 : xf+yg+zh=0 \}$. Note that, if $V$ did extend to a bundle on $S$, then every section of $V$ over $U$ would extend to a section of $S$ by Hartog, so $M$ would be a projective $k[x,y,z]$ module, but it isn't; this gives a proof that $V$ doesn't extend. $V$ doesn't quite do the job for us; we need $V \oplus T$, where $T$ is a trivial bundle. But that doesn't extend either for the same reason: If it did, then $M \oplus k[x,y,z]$ would be a projective $k[x,y,z]$ module, but then its summand $M$ would be, a contradiction.

Now, let $V^{\ast}$ be the dual bundle to $V$. Let $\mathcal{I}$ be the ideal sheaf of the zero section of $V^{\ast} \to U$ and set $X = \mathcal{O}_{V^{\ast}}/\mathcal{I}^2$. Then $f_{\ast} \cO_X \cong \mathcal{V} \oplus \cO_U$, where $\mathcal{V}$ is the sheaf of sections of $V$. In other words, $f_{\ast} \cO_X$ is the sheaf of sections of $V \oplus T$.

I now prove that, if the vector bundle extends, then so does $X$.

Set $Z = S \setminus U$ and assume that $f_{\ast} \cO_X$ extends to a vector bundle on $S$. I'll write $E$ for the total space of the bundle and $\cE$ for the sheaf of sections.

Extending in codimension $1$ We note that $X$ embeds in $E^{\ast}$ (the dual bundle) such that $f$ is the coordinate projection. The way this works is the following: Let $\mathcal{R}$ be the graded $\cO_U$-algebra $\cO_U \oplus (f_{\ast} \cO_X) \oplus (f_{\ast} \cO_X) \oplus \cdots$. Then $\mathrm{Proj} \mathcal{R} \cong X$. There is a map of $\cO_S$ algebras $\mathrm{Sym}^{\bullet} \cE \to \mathcal{R}$, which gives an embedding $X \to \mathbb{P}(E^{\ast})$. (Vakil, Exercise 7.3.J uses this trick to show finite maps are projective.)

So we get a map from $U$ to the Hilbert scheme of $\mathbb{P}(E^{\ast})/S$. Hilbert schemes are proper, and $S$ is smooth, so this map extends in codimension $1$. Thus, we may assume that $\mathrm{codim} Z \geq 2$.

Beyond codimension $1$ Now assume that $\mathrm{codim} Z \geq 2$. In this case, the extension of $f_{\ast} \cO_X$ to a vector bundle on $S$ can occur in at most one way.

The multiplicative identity of $\cO_X$ gives an section of $E$ over $U$. By Hartog's theorem, this section uniquely extends to a section $e: S \to E$. Similarly, the multiplication map $\mu : \cO_X \otimes_{\cO_S} \cO_X \to \cO_X$ gives a section of $\mathrm{Hom}(E \otimes E, E)$ over $U$, which extends uniquely to a section $\mu$ over $S$. The condition that $\mu$ and $e$ define the structure of a unital commutative algebra on $\cE$ is a closed condition, so they do. This uniquely equips $\cE$ with the structure of a (commutative, unital) $\cO_S$ algebra, and (relative) Spec of that algebra gives your $Y$.

$\def\cO{\mathcal{O}}\def\cE{\mathcal{E}}$I will show that the answer to (1) is yes if and only if the vector bundle $f_{\ast} \cO_X$ on $U$ extends to a vector bundle on $S$. Moreover, if $S \setminus U$ is codimension $\geq 2$ in $S$, I will show that the extension is unique as well.

This condition is obviously necessary.

Here is an example of a case where $f_{\ast} \cO_X$ does not extend. Start with Steven Landsburg's example (well, Swan's example, but Landsburg is easier to cite) of a vector bundle $V$ on $U:=\mathbb{A}^3 \setminus \{ 0 \}$ which doesn't extend $S:=\mathbb{A}^3$. If we take global sections of $V$ over $U$, we get $M:= \{ (f,g,h) \in k[x,y,z]^3 : xf+yg+zh=0 \}$. Note that, if $V$ did extend to a bundle on $S$, then every section of $V$ over $U$ would extend to a section of $S$ by Hartog, so $M$ would be a projective $k[x,y,z]$ module, but it isn't; this gives a proof that $V$ doesn't extend. $V$ doesn't quite do the job for us; we need $V \oplus T$, where $T$ is a trivial bundle. But that doesn't extend either for the same reason: If it did, then $M \oplus k[x,y,z]$ would be a projective $k[x,y,z]$ module, but then its summand $M$ would be, a contradiction.

Now, let $V^{\ast}$ be the dual bundle to $V$. Let $\mathcal{I}$ be the ideal sheaf of the zero section of $V^{\ast} \to U$ and set $X = \mathcal{O}_{V^{\ast}}/\mathcal{I}^2$. Then $f_{\ast} \cO_X \cong \mathcal{V} \oplus \cO_U$, where $\mathcal{V}$ is the sheaf of sections of $V$. In other words, $f_{\ast} \cO_X$ is the sheaf of sections of $V \oplus T$.

I now prove that, if the vector bundle extends, then so does $X$.

Set $Z = S \setminus U$ and assume that $f_{\ast} \cO_X$ extends to a vector bundle on $S$. I'll write $E$ for the total space of the bundle and $\cE$ for the sheaf of sections.

Extending in codimension $1$ We note that $X$ embeds in $E^{\ast}$ (the dual bundle) such that $f$ is the coordinate projection. The way this works is the following: Let $\mathcal{R}$ be the graded $\cO_U$-algebra $\cO_U \oplus (f_{\ast} \cO_X) \oplus (f_{\ast} \cO_X) \oplus \cdots$. Then $\mathrm{Proj} \mathcal{R} \cong X$. There is a map of $\cO_S$ algebras $\mathrm{Sym}^{\bullet} \cE \to \mathcal{R}$, which gives an embedding $X \to \mathbb{P}(E^{\ast})$. (Vakil, Exercise 7.3.J uses this trick to show finite maps are projective.)

So we get a map from $U$ to the Hilbert scheme of $\mathbb{P}(E^{\ast})/S$. Hilbert schemes are proper, and $S$ is smooth, so this map extends in codimension $1$. Thus, we may assume that $\mathrm{codim} Z \geq 2$.

Beyond codimension $1$ Now assume that $\mathrm{codim} Z \geq 2$. In this case, the extension of $f_{\ast} \cO_X$ to a vector bundle on $S$ can occur in at most one way.

The multiplicative identity of $\cO_X$ gives an section of $E$ over $U$. By Hartog's theorem, this section uniquely extends to a section $e: S \to E$. Similarly, the multiplication map $\mu : \cO_X \otimes_{\cO_S} \cO_X \to \cO_X$ gives a section of $\mathrm{Hom}(E \otimes E, E)$ over $U$, which extends uniquely to a section $\mu$ over $S$. The condition that $\mu$ and $e$ define the structure of a unital commutative algebra on $\cE$ is a closed condition, so they do. This uniquely equips $\cE$ with the structure of a (commutative, unital) $\cO_S$ algebra, and (relative) Spec of that algebra gives your $Y$.

$\def\cO{\mathcal{O}}\def\cE{\mathcal{E}}$I will show that the answer to (1) is yes if and only if the vector bundle $f_{\ast} \cO_X$ on $U$ extends to a vector bundle on $S$. Moreover, if $S \setminus U$ is codimension $\geq 2$ in $S$, I will show that the extension is unique as well.

This condition is obviously necessary. I do not yet have an example where $f_{\ast} \cO_X$ does not extend to a vector bundle. There are plenty of examples of vector bundles which don't extend from, for example, $\mathbb{A}^3 \setminus \{ 0 \}$ to $\mathbb{A}^3$. See Horrocks or Steven Landsburg's answer. But I haven't found one that supports a ring structure yet. Still, I would be pretty supposed if such an example couldn't be found.

Set $Z = S \setminus U$ and assume that $f_{\ast} \cO_X$ extends to a vector bundle on $S$. I'll write $E$ for the total space of the bundle and $\cE$ for the sheaf of sections.

Extending in codimension $1$ We note that $X$ embeds in $E^{\ast}$ (the dual bundle) such that $f$ is the coordinate projection. The way this works is the following: Let $\mathcal{R}$ be the graded $\cO_U$-algebra $\cO_U \oplus (f_{\ast} \cO_X) \oplus (f_{\ast} \cO_X) \oplus \cdots$. Then $\mathrm{Proj} \mathcal{R} \cong X$. There is a map of $\cO_S$ algebras $\mathrm{Sym}^{\bullet} \cE \to \mathcal{R}$, which gives an embedding $X \to \mathbb{P}(E^{\ast})$. (Vakil, Exercise 7.3.J uses this trick to show finite maps are projective.)

So we get a map from $U$ to the Hilbert scheme of $\mathbb{P}(E^{\ast})/S$. Hilbert schemes are proper, and $S$ is smooth, so this map extends in codimension $1$. Thus, we may assume that $\mathrm{codim} Z \geq 2$.

Beyond codimension $1$ Now assume that $\mathrm{codim} Z \geq 2$. The multiplicative identity of $\cO_X$ gives an section of $E$ over $U$. By Hartog's theorem, this section uniquely extends to a section $e: S \to E$. Similarly, the multiplication map $\mu : \cO_X \otimes_{\cO_S} \cO_X \to \cO_X$ gives a section of $\mathrm{Hom}(E \otimes E, E)$ over $U$, which extends uniquely to a section $\mu$ over $S$. The condition that $\mu$ and $e$ define the structure of a unital commutative algebra on $\cE$ is a closed condition, so they do. This uniquely equips $\cE$ with the structure of a (commutative, unital) $\cO_S$ algebra, and (relative) Spec of that algebra gives your $Y$.

$\def\cO{\mathcal{O}}\def\cE{\mathcal{E}}$I will show that the answer to (1) is yes if and only if the vector bundle $f_{\ast} \cO_X$ on $U$ extends to a vector bundle on $S$. Moreover, if $S \setminus U$ is codimension $\geq 2$ in $S$, I will show that the extension is unique as well.

This condition is obviously necessary.

Here is an example of a case where $f_{\ast} \cO_X$ does not extend. Start with Steven Landsburg's example (well, Swan's example, but Landsburg is easier to cite) of a vector bundle $V$ on $U:=\mathbb{A}^3 \setminus \{ 0 \}$ which doesn't extend $S:=\mathbb{A}^3$. If we take global sections of $V$ over $U$, we get $M:= \{ (f,g,h) \in k[x,y,z]^3 : xf+yg+zh=0 \}$. Note that, if $V$ did extend to a bundle on $S$, then every section of $V$ over $U$ would extend to a section of $S$ by Hartog, so $M$ would be a projective $k[x,y,z]$ module, but it isn't; this gives a proof that $V$ doesn't extend. $V$ doesn't quite do the job for us; we need $V \oplus T$, where $T$ is a trivial bundle. But that doesn't extend either for the same reason: If it did, then $M \oplus k[x,y,z]$ would be a projective $k[x,y,z]$ module, but then its summand $M$ would be, a contradiction.

Now, let $V^{\ast}$ be the dual bundle to $V$. Let $\mathcal{I}$ be the ideal sheaf of the zero section of $V^{\ast} \to U$ and set $X = \mathcal{O}_{V^{\ast}}/\mathcal{I}^2$. Then $f_{\ast} \cO_X \cong \mathcal{V} \oplus \cO_U$, where $\mathcal{V}$ is the sheaf of sections of $V$. In other words, $f_{\ast} \cO_X$ is the sheaf of sections of $V \oplus T$.

I now prove that, if the vector bundle extends, then so does $X$.

Set $Z = S \setminus U$ and assume that $f_{\ast} \cO_X$ extends to a vector bundle on $S$. I'll write $E$ for the total space of the bundle and $\cE$ for the sheaf of sections.

Extending in codimension $1$ We note that $X$ embeds in $E^{\ast}$ (the dual bundle) such that $f$ is the coordinate projection. The way this works is the following: Let $\mathcal{R}$ be the graded $\cO_U$-algebra $\cO_U \oplus (f_{\ast} \cO_X) \oplus (f_{\ast} \cO_X) \oplus \cdots$. Then $\mathrm{Proj} \mathcal{R} \cong X$. There is a map of $\cO_S$ algebras $\mathrm{Sym}^{\bullet} \cE \to \mathcal{R}$, which gives an embedding $X \to \mathbb{P}(E^{\ast})$. (Vakil, Exercise 7.3.J uses this trick to show finite maps are projective.)

So we get a map from $U$ to the Hilbert scheme of $\mathbb{P}(E^{\ast})/S$. Hilbert schemes are proper, and $S$ is smooth, so this map extends in codimension $1$. Thus, we may assume that $\mathrm{codim} Z \geq 2$.

Beyond codimension $1$ Now assume that $\mathrm{codim} Z \geq 2$. In this case, the extension of $f_{\ast} \cO_X$ to a vector bundle on $S$ can occur in at most one way. 

The multiplicative identity of $\cO_X$ gives an section of $E$ over $U$. By Hartog's theorem, this section uniquely extends to a section $e: S \to E$. Similarly, the multiplication map $\mu : \cO_X \otimes_{\cO_S} \cO_X \to \cO_X$ gives a section of $\mathrm{Hom}(E \otimes E, E)$ over $U$, which extends uniquely to a section $\mu$ over $S$. The condition that $\mu$ and $e$ define the structure of a unital commutative algebra on $\cE$ is a closed condition, so they do. This uniquely equips $\cE$ with the structure of a (commutative, unital) $\cO_S$ algebra, and (relative) Spec of that algebra gives your $Y$.