Geometric invariant theory works well when the algebraic group $G$ acting on a variety is reductive. There has been recent work by Doran and Kirwan here and here to find a canonical method of constructing GIT quotients for nonreductive groups. My question is what are potential applications for their work? One specific application they mention is constructing moduli of hypersurfaces in toric varieties. I would be interested in knowing of other applications.

On 80's Atiyah conference Kirwan spoke about one application. Namely she stated in her talk that there is an application to GreenGriffits conjecture. You can download the talk here http://www.maths.ed.ac.uk/~aar/atiyah80.htm and the slides are here http://www.icms.org.uk/downloads/GandP/Kirwan.pdf I am not sure if this was written down somewhere. 


This isn't really an answer, but I think it will help point in a useful direction. In case it's already obvious to you, please disregard. :) In linear algebra we classify $n\times n$ matrices up to conjugation, i.e. the $GL(n)$orbits on the set ${\mathfrak gl}(n)$ of $n\times n$ matrices. If we work over ${\mathbb C}$, then we know the complete classification of orbits by Jordan canonical form; and the $GL(n)$equivariant geometry of ${\mathfrak gl}(n)$ is beautiful and interesting. In particular, using Jordan form we know that every matrix can be conjugated to an upper triangular one; let ${\mathfrak b}$ be the set of upper triangular matrices. You might then ask, if we only allow ourselves to conjugate by the group $B$ of upper triangular matrices, what's the orbit structure now? There are reasons that this question is interesting. The first thing you might want to do is understand the function theory, i.e. the ring of invariant functions ${\mathbb C}[{\mathfrak b}]^B$, or more generally semiinvariant functions: i.e., choosing a character $\chi: B\rightarrow {\mathbb C}^*$, functions $f(b)$ for which $f(g\cdot b) = \chi(g)f(b)$ for all $g\in B$, $b\in {\mathfrak b}$. This leads to GIT for the nonreductive group $B$. Now, in this case it's not really necessary to deal with $B$, of course: we can replace $B$ by $G=GL(n)$ and ${\mathfrak b}$ by $\widetilde{g} = G\times_B {\mathfrak b}$, the GrothendieckSpringer resolution of ${\mathfrak gl}(n)$, and do GIT for the $GL(n)$action on $\widetilde{g}$. [And in fact, if I'm not mistaken, this method of inducing up to a reductive group and studying invariant theory for that larger group plays a role in the DoranKirwan theory?] Still, though, you can imagine situations that arise in nature in which you are interested in objects that naturally have some kind of filtration, and then the group by which you are quotienting just won't be reductive. These kinds of things arise naturally in studying moduli of decorated sheaves: see for example this paper of Drezet and Trautmann for the kind of thing that happens. 


I know of one potential application. Hain has some work described here http://arxiv.org/pdf/0802.0814 that deals with certain nonreductive GIT quotients that contain useful information about 3manifold invariants. The set of 3manifolds equipped with a genus g heegard splitting is naturally identified with the double coset space $H_g \backslash \Gamma_g / H_g$ where $\Gamma_g$ is the mapping class group of a genus g surface and $H_g \subset \Gamma_g$ is the handlebody subgroup. Using relative Malcev completions one replaces this with a double coset in which the subgroup is not reductive. So to construct the quotient one can try to use nonreductive GIT techniques. Last I heard, Doran and Hain were collaborating to work this out. 

