In a more positive direction, if the special fiber $X_s$ is not uniruled, then $X_s$ is birational to $Y_s$.
This can be found in a paper of Ulf Persson (memoirs of AMS 1977) perhaps over complex numbers. But the idea works over excellent rings (e.g. localization of finitely generated $\mathbb Z$-algebras): consider the graph $\Gamma$ of the birational map from $X$ to $Y$ and normalize it. By a theorem of Abhyankar, any irreducible component of the exceptional locus of $\Gamma\to Y$ is ruled, hence doesn't dominate $X_s$. Therefore $\Gamma\to X$ and $\Gamma\to Y$ are isomorphic over the generic points of $X_s$ and $Y_s$. So $X_s$ is birational to $Y_s$.