Suppose $S$ a noetherian base scheme, $X \to S$ is projective and $F, G$ are coherent $\mathcal O_X$-modules. Then by EGA (7.7.8) and (7.7.9) there exists a scheme $H = \underline{\operatorname{Hom}}_X(F, G)$, affine over $S$, which represents the functor $$(f: T \to S) \mapsto \operatorname{Hom}_{X_T}(f^*F, f^*G).$$ On $X_H$ there is a universal homomorphism $\varphi: F_H \to G_H$. Considering the exact sequence $$0 \to \ker \varphi \to F_H \xrightarrow{\varphi} G_H \to \operatorname{coker} \varphi \to 0,$$ one can check that the open set $$\underline{\operatorname{Isom}}_X(F, G):= H \setminus (\operatorname{supp}(\ker \varphi) \cup \operatorname{supp}(\operatorname{coker} \varphi)) \subset H$$ represents the functor $$(f: T \to S) \mapsto \operatorname{Isom}_{X_T}(f^*F, f^*G).$$ In [1, Section 2.1], Jason Starr and Johan de Jong write that $\underline{\operatorname{Isom}}_X(F, G)$ is affine over the base $S$. Why is that true? In general, opens of affine schemes are not affine.

[1] Jason Starr, Johan de Jong; *Almost proper GIT stacks and Discriminant Avoidance*

sheaf, not a scheme. And indeed there is a flatness assumption on $\mathscr{F}$. In any case, taking $X=S$, $\mathscr{F}=\mathscr{O}_X$, and $\mathscr{G}=$ a skyscraper sheaf at a closed point, I am sure there is no scheme $H$ as in the question. $\endgroup$4more comments