$\def\cO{\mathcal{O}}\def\cE{\mathcal{E}}$I will show that the answer to (1) is yes if and only if the vector bundle $f_{\ast} \cO_X$ on $U$ extends to a vector bundle on $S$. Moreover, if $S \setminus U$ is codimension $\geq 2$ in $S$, I will show that the extension is unique as well.

This condition is obviously necessary. I do not yet have an example where $f_{\ast} \cO_X$ does not extend to a vector bundle. There are plenty of examples of vector bundles which don't extend from, for example, $\mathbb{A}^3 \setminus \{ 0 \}$ to $\mathbb{A}^3$. See [Horrocks][1] or [Steven Landsburg's answer][2]. But I haven't found one that supports a ring structure yet. Still, I would be pretty supposed if such an example couldn't be found.

Set $Z = S \setminus U$ and assume that $f_{\ast} \cO_X$ extends to a vector bundle on $S$. I'll write $E$ for the total space of the bundle and $\cE$ for the sheaf of sections.


<b>Extending in codimension $1$</b> We note that $X$ embeds in $E^{\ast}$ (the dual bundle) such that $f$ is the coordinate projection. The way this works is the following: Let $\mathcal{R}$ be the graded $\cO_U$-algebra $\cO_U \oplus (f_{\ast} \cO_X) \oplus (f_{\ast} \cO_X) \oplus \cdots$. Then $\mathrm{Proj} \mathcal{R} \cong X$. There is a map of $\cO_S$ algebras $\mathrm{Sym}^{\bullet} \cE \to \mathcal{R}$, which gives an embedding $X \to \mathbb{P}(E^{\ast})$. ([Vakil][3], Exercise 7.3.J uses this trick to show finite maps are projective.)

So we get a map from $U$ to the Hilbert scheme of $\mathbb{P}(E^{\ast})/S$. Hilbert schemes are proper, and $S$ is smooth, so this map extends in codimension $1$. Thus, we may assume that $\mathrm{codim} Z \geq 2$.

<b>Beyond codimension $1$</b> Now assume that $\mathrm{codim} Z \geq 2$. The multiplicative identity of $\cO_X$ gives an section of $E$ over $U$. By Hartog's theorem, this section uniquely extends to a section $e: S \to E$. Similarly, the multiplication map $\mu : \cO_X \otimes_{\cO_S} \cO_X \to \cO_X$ gives a section of $\mathrm{Hom}(E \otimes E, E)$ over $U$, which extends uniquely to a section $\mu$ over $S$. The condition that $\mu$ and $e$ define the structure of a unital commutative algebra on $\cE$ is a closed condition, so they do. This uniquely equips $\cE$ with the structure of a (commutative, unital) $\cO_S$ algebra, and (relative) Spec of that algebra gives your $Y$.


  [1]: http://www.ams.org/mathscinet-getitem?mr=169877
  [2]: http://mathoverflow.net/a/92292/297
  [3]: http://math.stanford.edu/~vakil/216blog/FOAGapr2915public.pdf