Carving out subsheaves of local hom-sheaves of stacks of categories

Question

Recall from my previous question the definition of a local hom-sheaf of a stack of categories. I am interested in stacks of categories such that the underlying stack of groupoids is a moduli stack.

In his very useful answer Angelo helped me to see how Grothendieck's result about representability of Hilbert functors lets us know that the local hom-sheaves described by a sheaf on $Sch/S$ sending $T \to S$ to the set of maps $T \times_S X \to T \times_S Y$ of $S$-schemes are representable (by schemes or algebraic spaces, depending on hypotheses). What I am interested in is the more general situation where the hom-sheaves are given by a subsheaf of this type of sheaf, namely one carved out by conditions on the maps, such as being homomorphisms of abelian varieties, perhaps preserving polarisations, or taking some or all of a set of marked points to a set of marked points, or more generally mapping a subscheme to a subscheme, or even some sort of stratified map preserving some natural stratification.

I know what I would do if I were in the more 'permissible' category of topological spaces, or perhaps even (Frechet) smooth manifolds: form some equaliser (or limit) of the spaces involved. But I do not have a good feeling of when this sort of thing is possible for schemes. My question is this

What sort of conditions on maps of schemes can I impose such that the resulting subsheaf of the representable local hom-sheaf described above is also representable?

Answer 1

Taking equalizers, or limits, is also the standard algebraic geometry way. So, for example, say that $X \to S$ and $Y \to S$ are finitely presented and proper, with $X\to S$ flat, and have sections $S \to X$ and $S \to Y$. You want to consider the subsheaf $H'$ of $H :=\underline{Hom}_S(X, Y)$ of morphisms preserving the section. There is a tautological morphism $f \colon H \times_S X \to H \times_S Y$. The two sections give you a sections $s_H\colon H\to H \times_S X$ and $t_H\colon H\to H \times_S Y$; your subsheaf $H'$ is the equalizer of $t_H$ and the composite $f\circ s_H\colon H \to H \times_S Y$.

As another example, suppose that you have closed subschemes $X'$ and $Y'$, with $X' \to S$ flat, and you want to consider the subsheaf of $H'$ of $H :=\underline{Hom}_S(X, Y)$ of morphism sending $X'$ inside $Y'$. The embeddings $X' \subseteq X$ and $Y' \subseteq Y$ induce morphisms $\underline{Hom}_S(X, Y) \to \underline{Hom}_S(X', Y)$ and $\underline{Hom}_S(X', Y') \to \underline{Hom}_S(X', Y)$, the second one being a monomorphism of sheaves (it is in fact a closed embedding of algebraic spaces, but this does not matter here). Then $H'$ is the fibered product $$ \underline{Hom}_S(X, Y) \times_{\underline{Hom}_S(X', Y)} \underline{Hom}_S(X', Y')\,. $$