MathOverflow will be down for maintenance for approximately 3 hours, starting Monday evening (06/24/2013) at approximately 9:00 PM Eastern time (UTC-4).
show/hide this revision's text 2 added 604 characters in body

As Torsten points out, unipotent groups correspond naturally to nilpotent Lie algebras in characteristic 0. This is dealt with nicely on the scheme level, for example, in IV.2.4 of Demazure-Gabriel Groupes algebriques. They also treat in Chapter IV some questions about prime characteristic, which get quite tricky outside the commutative case. Both of your questions are more conveniently studied in the Lie algebra framework, I think, where standard Lie algebra methods for discussing forms in are available and where there is quite a bit of literature on structure, representations, and (in small dimensions) classification in characteristic 0. See for example Jacobson's 1962 book Lie Algebras.

Representation theory is potentially very complicated for nilpotent Lie algebras (say over the complex or real field), even in the finite dimensional situation: unlike the semisimple case, there is no nice general structure based on highest weights, etc. Dixmier and others have studied infinite dimensional representations extensively in connection with Lie groups. Classification of nilpotent Lie algebras is just about impossible in general, but up to dimension 7 or so there are lists. Anyway, there is a lot of literature out there. (Tori are on the other hand also studied a lot over fields of interest in number theory. They have at least the advantage of being commmutative.)

[ADDED] An older seminar write-up might be worth consulting, especially in prime characteristic, along with the relatively sparse literature published since then and best searched through MathSciNet: Unipotent Algebraic Groups by T. Kambayashi, M. Miyanishi, M. Takeuchi, Springer Lecture Notes in Math. 414 (1974). But as their treatment suggests, the main research challenges have occurred in prime characteristic. Over finite fields, there has been quite a bit of recent activity in studying the characters of finite unipotent groups related to the unipotent radical of a Borel subgroup.

show/hide this revision's text 1

As Torsten points out, unipotent groups correspond naturally to nilpotent Lie algebras in characteristic 0. This is dealt with nicely on the scheme level, for example, in IV.2.4 of Demazure-Gabriel Groupes algebriques. They also treat in Chapter IV some questions about prime characteristic, which get quite tricky outside the commutative case. Both of your questions are more conveniently studied in the Lie algebra framework, I think, where standard Lie algebra methods for discussing forms in are available and where there is quite a bit of literature on structure, representations, and (in small dimensions) classification in characteristic 0. See for example Jacobson's 1962 book Lie Algebras.

Representation theory is potentially very complicated for nilpotent Lie algebras (say over the complex or real field), even in the finite dimensional situation: unlike the semisimple case, there is no nice general structure based on highest weights, etc. Dixmier and others have studied infinite dimensional representations extensively in connection with Lie groups. Classification of nilpotent Lie algebras is just about impossible in general, but up to dimension 7 or so there are lists. Anyway, there is a lot of literature out there. (Tori are on the other hand also studied a lot over fields of interest in number theory. They have at least the advantage of being commmutative.)