Invariants of the maximal unipotent subgroup of GL(n) acting on the space of n by n matrices

Question

Let $G=GL(n,\mathbb{C})$ and let $U\subset G$ be a maximal unipotent subgroup. (For example,assume that U is the set of upper triangular matrices with ones in the diagonal.) Now let $X=M_{n}(\mathbb{C})$ be the space of $n$ by $n$ complex matrices and consider the usual action of $U$ on $X$ given by left multiplication. Is it possible to describe the space $\mathbb{C}[X]^{U}$ explicitly using generators and relations? In other words, is there an explicit realization of the categorial quotient of $X$ by $U$ as an affine algebraic variety?

Answer 1

Here is a coordinate-free description of the affine hull of the quotient of $G=\text{Aut}_k(V)$ by the right action of the unipotent radical $U$ of a Borel subgroup $B$. The group $B$ is the stabilizer of a flag of $k$-linear subspaces, $$\{0\} = F^0 \subsetneqq F^1 \subsetneqq \dots \subsetneqq F^r \subsetneqq F^n = V.$$ For $r=1,\dots,n-1$, denote by $H^r$ the affine $k$-space with underlying $k$-vector space $\text{Hom}_k(\bigwedge^r F^r,\bigwedge^r V)$. Denote by $Z^r\subset H^r$ the affine cone over the Grassmannian $\text{Grass}(r,V)$, i.e., the homogeneous, Zariski closed subset parameterizing all linear maps of the form $\bigwedge^r T^r$ for a $k$-linear morphism $$T^r:F^r \to V.$$ The defining ideal of $Z^r$ in the (polynomial) coordinate ring $k[H^r]$ is generated by explicit Plücker quadratic relations, cf. Griffiths-Harris (for instance). Denote by $Z^n$ the open complement of the origin in the $1$-dimensional affine space $\text{Hom}_k(\bigwedge^n F^n, \bigwedge^n V) =\text{End}_k(\bigwedge^n V)$.

Inside $Z^1 \times_k Z^2 \times_k \dots \times_k Z^{n-1}$, let $Z'$ be the affine cone over the flag variety $\text{Flag}(1,2,\dots,n-1;V)$. This is cut out by explicit equations similar to the Plücker relations. For instance, for every $r=2,\dots,n-1$, one relation on $(z^1,\dots,z^{n-1})$ is that $z^1\wedge z^r$ equals $0$ as an element in $\text{Hom}_k(\bigwedge^1F^1\otimes_k \bigwedge^r F^r,\bigwedge^{r+1}V)$. I do realize that you want an explicit list of relations, and I am not quite giving these. However, you can quickly find the explicit relations by searching for "Cox ring" and "flag variety".

Anyway, the affine hull of the quotient $G/U$ of the right action of $U$ on $G$ is $Z = Z'\times Z^n$. The quotient morphism is $$q: G \to Z, \ \ q(T) = \left( \bigwedge^1(T|_{F^1}), \bigwedge^2(T|_{F^2}),\dots, \bigwedge^{n-1}(T|_{F^{n-1}}),\bigwedge^n(T|_{F^n}) \right).$$ Although $Z$ is affine, the morphism $q$ is not surjective. In particular, an element $(z^1,\dots,z^{n-1},z^n)$ is in the image if and only if $z^r$ is nonzero for every $r=1,\dots,n-1$. This is the point I was making in my comment above.

Edit. I checked a textbook on the combinatorics of these kinds of commutative rings.

Combinatorial Commutative Algebra
Ezra Miller, Bernd Sturmfels
Graduate Texts in Mathematics (Book 227), 420 pages.
Springer.
ISBN-10: 0387237070

On p. 275, Definition 14.2, they define the ring $k[Z']$ to be the Plücker algebra. It is generated by the linear bases for the dual vector spaces of $H^1,\dots,H^{n-1}$, which total $2^n-2$ variables. The ideal of relations is generated in degree $2$. Most of Chapter 14 of that book is devoted to describing these quadratic relations.

Answer 2

Questions of this type can be approached from the direction of algebraic geometry (in a suitable generality) or from the direction of invariant theory involving algebraic group actions. I'd point especially to Chapter 3 in the 1997 monograph by Frank D. Grosshans, Algebraic homogeneous spaces and invariant theory, Lect. Notes in Math. 1673, Springer. He works in some generality throughout, taking $k$ to be any algebraically closed field of any characteristic and using "traditional" algebraic geometry language. In particular, he looks at the case of general linear groups in $\S13$ of that chapter and describes (for either left or right action) the rings of invariants for $U$ on suitable algebras of polynomials.

Grosshans builds on older foundations for reductive group actions on varieties and resulting homogeneous spaces, going back to Hilbert, Mumford, and others. His results rely a lot on combinatorics and are often more comprehensive than what you ask for. On the other hand, he is fairly straightforward and careful about the details. Though he works in arbitrary characteristic, his approach is not too different from that of the Russian school (Vinberg, Popov, Panyushev, ...) who typically work over an algebraically closed field of characteristic 0. Along the way, Grosshans does provide a fairly detailed concrete picture of the quotients obtained, starting with the generators of rings of invariants.