Let $G=GL(n,\mathbb{C})$ and let $U\subset G$ be a maximal unipotent subgroup. (For example,assume that U is the set of upper triangular matrices with ones in the diagonal.) Now let $X=M_{n}(\mathbb{C})$ be the space of $n$ by $n$ complex matrices and consider the usual action of $U$ on $X$ given by left multiplication. Is it possible to describe the space $\mathbb{C}[X]^{U}$ explicitly using generators and relations? In other words, is there an explicit realization of the categorial quotient of $X$ by $U$ as an affine algebraic variety?

$\begingroup$ Just for clarification: is $\mathbb{C}[X]$ the polynomial ring in $n^2$ variables? (It can be the group algebra as well...) And $\mathbb{C}[X]^U$ is the invariants under $U$? $\endgroup$ – WhatsUp Jul 23 '15 at 21:37

$\begingroup$ @WhatsUp Yes, it is the polynomial algebra. $\endgroup$ – Raul Gomez Jul 23 '15 at 21:46

$\begingroup$ When you ask about the "categorical quotient", in what category, precisely, are you working? The categorical quotient in the category of schemes is not an affine scheme (just consider the case when $n$ equals $2$). You might want to look up "Matsushima's criterion" (extended to positive characteristic by Richardson). $\endgroup$ – Jason Starr Jul 23 '15 at 23:07

$\begingroup$ @Jason Sorry I was imprecise. What I'm looking for is a description of $\operatorname{Spec} \mathbb{C}[X]^{U}$ $\endgroup$ – Raul Gomez Jul 24 '15 at 2:43

$\begingroup$ @RaulGomez. "What I'm looking for is a description of $\text{Spec} \ \mathbb{C}[X]^U$." Are you looking for a description as a subvariety $Z$ of an affine space? I gave that below: up to the $\mathbb{G}_m$factor $Z^n$, $Z$ is just the affine cone over the flag variety. $\endgroup$ – Jason Starr Jul 24 '15 at 2:50
Here is a coordinatefree description of the affine hull of the quotient of $G=\text{Aut}_k(V)$ by the right action of the unipotent radical $U$ of a Borel subgroup $B$. The group $B$ is the stabilizer of a flag of $k$linear subspaces, $$\{0\} = F^0 \subsetneqq F^1 \subsetneqq \dots \subsetneqq F^r \subsetneqq F^n = V.$$ For $r=1,\dots,n1$, denote by $H^r$ the affine $k$space with underlying $k$vector space $\text{Hom}_k(\bigwedge^r F^r,\bigwedge^r V)$. Denote by $Z^r\subset H^r$ the affine cone over the Grassmannian $\text{Grass}(r,V)$, i.e., the homogeneous, Zariski closed subset parameterizing all linear maps of the form $\bigwedge^r T^r$ for a $k$linear morphism $$T^r:F^r \to V.$$ The defining ideal of $Z^r$ in the (polynomial) coordinate ring $k[H^r]$ is generated by explicit Plücker quadratic relations, cf. GriffithsHarris (for instance). Denote by $Z^n$ the open complement of the origin in the $1$dimensional affine space $\text{Hom}_k(\bigwedge^n F^n, \bigwedge^n V) =\text{End}_k(\bigwedge^n V)$.
Inside $Z^1 \times_k Z^2 \times_k \dots \times_k Z^{n1}$, let $Z'$ be the affine cone over the flag variety $\text{Flag}(1,2,\dots,n1;V)$. This is cut out by explicit equations similar to the Plücker relations. For instance, for every $r=2,\dots,n1$, one relation on $(z^1,\dots,z^{n1})$ is that $z^1\wedge z^r$ equals $0$ as an element in $\text{Hom}_k(\bigwedge^1F^1\otimes_k \bigwedge^r F^r,\bigwedge^{r+1}V)$. I do realize that you want an explicit list of relations, and I am not quite giving these. However, you can quickly find the explicit relations by searching for "Cox ring" and "flag variety".
Anyway, the affine hull of the quotient $G/U$ of the right action of $U$ on $G$ is $Z = Z'\times Z^n$. The quotient morphism is $$q: G \to Z, \ \ q(T) = \left( \bigwedge^1(T_{F^1}), \bigwedge^2(T_{F^2}),\dots, \bigwedge^{n1}(T_{F^{n1}}),\bigwedge^n(T_{F^n}) \right).$$ Although $Z$ is affine, the morphism $q$ is not surjective. In particular, an element $(z^1,\dots,z^{n1},z^n)$ is in the image if and only if $z^r$ is nonzero for every $r=1,\dots,n1$. This is the point I was making in my comment above.
Edit. I checked a textbook on the combinatorics of these kinds of commutative rings.
Combinatorial Commutative Algebra
Ezra Miller, Bernd Sturmfels
Graduate Texts in Mathematics (Book 227), 420 pages.
Springer.
ISBN10: 0387237070
On p. 275, Definition 14.2, they define the ring $k[Z']$ to be the Plücker algebra. It is generated by the linear bases for the dual vector spaces of $H^1,\dots,H^{n1}$, which total $2^n2$ variables. The ideal of relations is generated in degree $2$. Most of Chapter 14 of that book is devoted to describing these quadratic relations.

$\begingroup$ I just have a small question regarding your answer. It seems to me that you should be able to extend the morphism $q$ that you are defining above to the whole of $\operatorname{End}_{k}(V)$. Is this extended map then surjective? $\endgroup$ – Raul Gomez Jul 24 '15 at 23:11

$\begingroup$ @RaulGomez: "Is this extended map then surjective?" No, the extended map is not surjective. For instance, the element $(q^1,q^2,\dots,q^{n1},q^n) = (0,0,\dots, 0,1)$ is not in the image. $\endgroup$ – Jason Starr Jul 25 '15 at 0:00
Questions of this type can be approached from the direction of algebraic geometry (in a suitable generality) or from the direction of invariant theory involving algebraic group actions. I'd point especially to Chapter 3 in the 1997 monograph by Frank D. Grosshans, Algebraic homogeneous spaces and invariant theory, Lect. Notes in Math. 1673, Springer. He works in some generality throughout, taking $k$ to be any algebraically closed field of any characteristic and using "traditional" algebraic geometry language. In particular, he looks at the case of general linear groups in $\S13$ of that chapter and describes (for either left or right action) the rings of invariants for $U$ on suitable algebras of polynomials.
Grosshans builds on older foundations for reductive group actions on varieties and resulting homogeneous spaces, going back to Hilbert, Mumford, and others. His results rely a lot on combinatorics and are often more comprehensive than what you ask for. On the other hand, he is fairly straightforward and careful about the details. Though he works in arbitrary characteristic, his approach is not too different from that of the Russian school (Vinberg, Popov, Panyushev, ...) who typically work over an algebraically closed field of characteristic 0. Along the way, Grosshans does provide a fairly detailed concrete picture of the quotients obtained, starting with the generators of rings of invariants.