How is the sheaf defined for $G/H$ where $G$ is an algebraic group and $H$ is a normal closed subgroup?

Question

I have learned that if $G$ is an algebraic group and $H$ is a normal closed subgroup then $G/H$ is also an algebraic group satisfying: for any morphisms $\phi : G \rightarrow X$ constant on the classes $gH$, there exists a unique morphism $\psi: G/H \rightarrow X$ such that $\phi = \psi \circ \pi$, where $\pi: G \rightarrow G/H$ is the quotient.

How is the sheaf on $G/H$ defined? (and how is it defined on the open set $U$ say?)

Any information would be appreciated. Thank you very much.

Answer 1

Since the OP added a follow-up question in the comments, I am writing up the comments by Moret-Bailly and myself as an answer.

For a base scheme $S$, e.g., $S=\text{Spec}\ k$ for $k$ a field or $S=\text{Spec}\ \mathbb{Z}$. There are various Grothendieck topologies on the category of $S$-schemes. Since many theorems about $S$-group schemes are stated for faithfully flat, finitely presented $S$-group schemes, it is convenient to work with the Grothendieck topology in which covering families are collections of morphisms to a fixed target whose disjoint union morphism to that target is fppf, faithfully flat and finitely presented.

For an $S$-group scheme $G$, for an $S$-scheme $X$, and for an $S$-action of $G$ on $X$, $$\rho:G\times_S X \to X,$$ a $\rho$-invariant $S$-morphism is an $S$-morphism $$q:X\to Y,$$ such that the following two composite $S$-morphisms are both equal, $$G\times_S X \xrightarrow{\rho} X \xrightarrow{q} Y, \ \ G\times_S X \xrightarrow{\text{pr}_2} X \xrightarrow{q} Y.$$ Call this common morphism $q_G$. A categorical quotient of the $S$-action $\rho$ on $X$ is a $\rho$-invariant $S$-morphism, $p:X\to Z$, that is initial among all $\rho$-invariant $S$-morphisms, i.e., for every $\rho$-invariant $S$-morphism $q:X\to Y$ there exists a unique $S$-morphism $f:Z\to Y$ such that $q$ equals $f\circ p$. In particular, by considering morphisms from $X$ to $\mathbb{A}^1_S$, for every categorical quotient, the pullback homomorphism, $$\mathcal{O}(p):\mathcal{O}_Z(Z)\to \mathcal{O}_X(X),$$ is injective with image equal to the $G$-invariant subring of $\mathcal{O}_X(X)$.

A uniform categorical quotient of the $S$-action $\rho$ on $X$ is an $S$-morphism, $p:X\to Z$, such that for every flat, finitely presented $S$-morphism $u:U\to Z$, for the fiber product $X_U:= X\times_Z U$, for the induced $S$-action of $G$ on $X_U$, $$\rho_U:(G\times_S X) \times_{q_G,Z,u} U \xrightarrow{\rho \times \text{Id}_U} X \times_{q,Z,u} U,$$ the following $\rho_U$-invariant $S$-morphism is a categorical quotient, $$\text{pr}_2: X\times_Z U \to U.$$ In particular, allowing $u$ to vary among all open immersions, the previous paragraph implies that for every uniform categorical quotient, the induced sheaf homomorphism, $$p^{\#}:\mathcal{O}_Z\to p_*\mathcal{O}_X,$$ is injective with image equal to the $G$-invariant subsheaf.

Finally, one of the main sources of uniform categorical quotients are fppf $G$-torsors. A $\rho$-invariant $S$-morphism $p$ is a fppf $G$-torsor if $p$ is fppf and the following induced $S$-morphism is an isomorphism, $$\Psi:G\times_S X \xrightarrow{(\rho,\text{pr}_2)} X\times_S X.$$ This condition implies that $\rho$ is a uniform categorical quotient, and thus $p^{\#}$ is injective with image equal to the $G$-invariant subsheaf of the pushforward of $\mathcal{O}_X$.

The following book of Raynaud examines in detail when a categorical quotient exists as an algebraic space, and when it exists as a quasi-projective scheme, for $S$ quite general.

MR0260758 (41 #5381)
Raynaud, Michel
Faisceaux amples sur les schémas en groupes et les espaces homogènes.
Lecture Notes in Mathematics, Vol. 119
Springer-Verlag, Berlin-New York 1970 ii+218 pp.

When $S$ equals $\text{Spec} \ k$, with $k$ a field, when $X$ is itself a finitely presented $k$-group scheme $G'$, and when the action $\rho$ is the regular action of a closed $k$-subgroup scheme of $G'$, then Chow proved that the categorical quotient exists as a quasi-projective $k$-scheme. If $G'$ is smooth, you can find a proof that the quotient is an fppf $G$-torsor as Proposition 0.9, p. 16 of the following.

MR1304906 (95m:14012)
Mumford, D.; Fogarty, J.; Kirwan, F.
Geometric invariant theory. Third edition.
Ergebnisse der Mathematik und ihrer Grenzgebiete (2), 34.
Springer-Verlag, Berlin, 1994. xiv+292 pp.

In particular, that applies to the case of the quotient of a smooth reductive group scheme by a parabolic subgroup scheme.

It is a little backward to suggest that the geometric quotient is constructed without first writing down a structure sheaf on the quotient. Nonetheless, the logic above does imply that in every such case, however the quotient is constructed, the structure sheaf of the quotient is canonically isomorphic to the $G$-invariant subsheaf of the pushforward of the structure sheaf of $X$. You can find more details in Mumford's "Geometric Invariant Theory".