I am reading a note at Page 63
ftp://ftp.math.ethz.ch/users/pink/FGS/CompleteNotes.pdf
It says whenever $G$ is finite and flat over $S$ the functor ${\rm Hom}(G,H)$ is representable. But it does not give the proof or any references.
Can anybody help me with this question?
The typical way statements like this are proven is by deducing them from the representability of Hilbert schemes. For example, we can use:
Theorem (Grothendieck). Let $S$ be a Noetherian scheme, let $X$ and $Y$ be $S$-schemes of finite type, and assume $X$ is flat and projective, and $Y$ is quasi-projective. Then the functor \begin{align*} \operatorname{\underline{Sch}}\!/S &\to \operatorname{\underline{Set}}\\ T &\mapsto \operatorname{Mor}_T(X_T,Y_T) \end{align*} is representable by a separated $S$-scheme $\operatorname{\underline{Mor}}(X,Y)$ that is a rising union of quasi-projective $S$-schemes.
Proof. See for example [FGA TDTE IV, §4c], or [FGAE, Thm. 5.23]. $\square$
Thus, if $G$ is finite and $H$ is quasiprojective, then this guarantees existence of $$\operatorname{\underline{Mor}}(G,H).$$ This is the functor of morphisms as schemes; to get the functor of morphisms of group schemes, just note that a map $f \colon G \to H$ is a morphism of group schemes if and only if the diagram $$\begin{array}{ccc}G \times G & \stackrel{f \times f}\longrightarrow & H \times H \\ \!\!\!\!{\scriptsize{\mu}}\downarrow & & \downarrow{\scriptsize{\mu}}\!\!\!\! \\ G & \stackrel{f}\longrightarrow & H\end{array}$$ commutes. Therefore, the homomorphisms of group schemes are given by the fibre product $$\begin{array}{ccc}\operatorname{\underline{Hom}}(G,H) & \longrightarrow & \operatorname{\underline{Mor}}(G,H) \\ \downarrow & & \downarrow \\ \operatorname{\underline{Mor}}(G \times G, H) & \stackrel{\Delta}\longrightarrow & \operatorname{\underline{Mor}}(G \times G, H) \underset S\times \operatorname{\underline{Mor}}(G \times G, H),\! \end{array}$$ where the right vertical morphism maps $f$ to $(\mu_H \circ f \times f, f \circ \mu_G)$. Since $\operatorname{\underline{Mor}}(G \times G, H)$ is separable, the diagonal is a closed immersion, hence the same goes for the natural inclusion $$\operatorname{\underline{Hom}}(G, H) \to \operatorname{\underline{Mor}}(G, H).$$ Therefore, $\operatorname{\underline{Hom}}(G,H)$ is representable by a separated $S$-scheme that is a rising union of quasi-projective $S$-schemes. $\square$
Remark. I don't know how strong an assumption the quasi-projectivity of $H$ is, but in most examples one cares about this is true. There are examples of abelian schemes that are not projective, but in that case there are other arguments that still show that $\operatorname{\underline{Hom}}(A, B)$ is representable by a scheme. It always works if you're willing to work with algebraic spaces; see for example [Tag 0D1C].
Exercise. If $G$ is a finite (constant) group, then show that $\operatorname{\underline{Hom}}(G, H)$ is a closed subset of a suitable power of $H$, hence is of finite type over $S$. (The proof I gave above a priori only gives an $S$-scheme that is locally of finite type.)
Hence, by a descent argument, the same holds when $G$ is étale-locally isomorphic to a finite constant group, or equivalently if $G \to S$ is finite étale. This gives an easier proof of representability in this case, but I don't know how to generalise this method to the general finite flat case. I'm not sure if $\operatorname{\underline{Hom}}(G,H)$ is always of finite type over $S$.
Exercise. Why is $\operatorname{\underline{Hom}}(\mathbb G_a, \mathbb G_m)$ not representable? (Hint: use your intuition over fields to write down what its set of points should be. Use more and more non-reduced schemes to show that there is no appropriate scheme structure.)
References.
[FGA TDTE IV] A. Grothendieck, Techniques de descente et théoremes d’existence en géométrie algébrique. IV: Les schemas de Hilbert, Sem. Bourbaki 13, No. 221 (1961). ZBL0236.14003.
[FGAE] B. Fantechi, L. Göttsche, L. Illusie, S. L. Kleiman, N. Nitsure, A. Vistoli, Fundamental algebraic geometry: Grothendieck’s FGA explained. Mathematical Surveys and Monographs 123. AMS, Providence, RI (2005). ZBL1085.14001.
You could look at Section 6.3 of this memoir: https://arxiv.org/pdf/math/0011121.pdf
I think that I probably first saw an argument of this type in Demazure and Gabriel's "Groupes algebriques". The technical context there is a little different, and might or might not be more useful for you, depending on what you want to do.
