Let $X\to B$ and $Y\to B$ be two flat, projective $B$-schemes ($S$ is already taken ;-). Then
Let $\mathscr Hom_B(X,Y)$ be the functor defined by $$\mathscr Hom_B(X,Y)(Z)=\{Z{\rm -morphisms }\ X\times_B Z\to Y\times_B Z\}.$$ where $Z\to B$ is also a $B$-scheme. Then $\mathscr Hom_B(X,Y)$ is represented by an open $B$-subscheme $${\rm Hom}_B(X,Y)\subset {\rm Hilb}_B(X\times_BY).$$
The $\mathscr Hom$ functor has a subfunctor $\mathscr Isom$ consisting of those morphisms that define a relative isomorphism. This is represented by an open subscheme $${\rm Isom}_B(X,Y)\subset {\rm Hom}_B(X,Y).$$
Now if $B$ is a point, $X=Y$, then this $\rm Isom$ scheme can be identified with the automorphism group of $X$.