Let $X\to B$ and $Y\to B$ be two flat, projective $B$-schemes ($S$ is already taken ;-). Then
Let $\mathscr Hom_B(X,Y)$ be the functor defined by $$\mathscr Hom_B(X,Y)(Z)=\{B{\rm -morphisms }\ X\times_B Z\to Y\times_B Z\}.$$ where $Z\to B$ is also a $B$-scheme. Then $\mathscr Hom_B(X,Y)$ is represented by an open $B$-subscheme $$Hom_B(X,Y)\subset Hilb_B(X\times_BY).$$
The $\mathscr Hom$ functor has a subfunctor $\mathscr Isom$ which is represented by an open subscheme $Isom\subset Hom$.
Now if $B$ is a point, $X=Y$, then this $Isom$ scheme can be identified with the automorphism group of $X$.