The title pretty much summarizes the question: does every $p$group have a faithful unipotent representation (with coefficients in $\mathbb{F}_p$ or some finite extension thereof)?

$\begingroup$ Infinite $p$groups such as Burnside groups for $p$ large enough are not linear. But I suppose the notion of unipotent representation could still make sense for an infinitedimensional representation, meaning that subtracting the identity matrix from any element gives a nilpotent matrix (some power is zero). $\endgroup$ – Ian Agol Feb 17 '11 at 17:19

$\begingroup$ I did ask the question about finite groups, but the infinite question is interesting also... I edited the title in the meantime. $\endgroup$ – Igor Rivin Feb 17 '11 at 20:52

$\begingroup$ And it seems that @Yuri is answering the infinite question also. $\endgroup$ – Igor Rivin Feb 17 '11 at 20:53
I guess you mean finite $p$groups. Then every finitedimensional representation of a finite $p$group $G$ is unipotent in characteristic $p$. (Indeed, all eigenvalues of every element of $G$ are $p$power roots of unity and therefore must be equal to $1$. This means that every element of $G$ acts as a unipotent operator.) An example of a faithful unipotent representation of $G$ is provided by its regular representation over any field of characteristic $p$. In particular, the regular representation of $G$ over the prime field $F_p$ is faithful and unipotent.
(The same construction works for infinite $p$groups $G$ as well. Of course, in this case the ``regular" representation space of functions on $G$ with finite support will be infinitedimensional.)

$\begingroup$ In the infinite case, maybe you mean $G$ prop, and continuous functions to $F_p$ with compact support? What is the $G$invariant subspace in your last example? $\endgroup$ – vytas Feb 18 '11 at 14:02

$\begingroup$ sorry the support is automatically compact. I mean $G$ prop, and the space of cts function to $F_p$. $\endgroup$ – vytas Feb 18 '11 at 14:04

$\begingroup$ Oh, no: I mean a $p$group, i.e., a periodic group where the order of every element is a power of $p$. $\endgroup$ – Yuri Zarhin Feb 18 '11 at 14:31

$\begingroup$ I see, i was confused by the meaning of unipotent. For me it means increasing exhaustive filtration by subreps, starting with $0$, such that $G$ acts trivially on the graded pieces. i guess you take decreasing filtration. $\endgroup$ – vytas Feb 18 '11 at 15:17

$\begingroup$ In characteristic $p$ there does not exist a faithful unipotent representation of the additive group $Z_p$ of $p$adic integers. Indeed, let $\rho: Z_p \to Aut(V)$ be such a representation and let $n>1$ be the smallest integer such that there exists a (nonzero) $a \in Z_p$ such that (nonzero) $N=(\rho(a)1)$ satisfies $N^n=0$. Then $\rho(pa)=(1+N)^p=1+N^p$. Let $m$ be the smallest positive integer that is greater or equal than $n/p$. Then $m<n$ and $(\rho(pa)1)^m=N^{mp}=0$. The minimality of $n$ implies that $m=1$, i.e., $\rho(pa)=1$, which contradicts the faithfulness of $\rho$. $\endgroup$ – Yuri Zarhin Feb 18 '11 at 15:41
Yes, of course, it is how Bogopolsky proves Sylow theorems in his book (Bogopolski, Oleg, Introduction to group theory. Translated, revised and expanded from the 2002 Russian original. EMS Textbooks in Mathematics. European Mathematical Society (EMS), Zürich, 2008. x+177 pp.).
More precisely, a finite $p$group $G$ of order $n$ embeds into $GL=GL_n({\mathbb F}_p)$ (Cayley theorem). The subgroup $UT$ of upper triangular unipotent matrices of $GL$ is a Sylow psubgroup of $GL$ (proof by computing the order of $GL$), so $G$ is conjugate to a subgroup of $UT$.

$\begingroup$ Thanks for the Bogopolski reference (and the answer, of course...) I assume you recommend the rest of the book, as well. I will try to get my hands on it... $\endgroup$ – Igor Rivin Feb 18 '11 at 2:23

1$\begingroup$ That is a very good book. I used it several times in introductory group theory courses. I think it is better than Rotman and KargapolovMerzlyakov basically because it is more modern and covers more interesting material in a very little space. As for the Sylov theorem, I have not seen conceptually better proofs anywhere else. Most probably it was invented many years ago but did not make it to a book till recently. On the other hand it is possible that Oleg invented the proof himself. $\endgroup$ – Mark Sapir Feb 18 '11 at 19:49
Here is a slightly different approach. If $G$ is a $p$group (infinite or finite) and $k$ is a field of characteristic $p$ then in the group algebra $k[G]$ we have $(x1)^{p^r}=x^{p^r}1$ for all $x \in k[G]$. (Actually, if elements $a$ and $b$ of any $k$algebra do commute then $(ab)^{p^r}=a^{p^r}  b^{p^r}$, thanks to divisibility properties of binomial coefficients.) Applying it to $x=g$ where $g$ is an element of $G$ of order $p^r$, we conclude that $(g1)^{p^r}=g^{p^r}1=11=0$ in $k[G]$. Since every representation space $V$ of $G$ over $k$ is a module over $k[G]$, we conclude that $g1$ acts on $V$ as a nilpotent operator.
An example of a faithful representation of $G$ is provided by the regular representation where $V$ is the space of all $k$valued functions on $G$. Another example is provided by its $G$invariant space $V_0$ of all functions $f: G \to k$ with finite support (i.e., vanishing at all but finitely many points of $G$). Notice that for each $g \in G$ the space $V_0$ splits into a direct sum of $g$invariant finitedimensional subspaces (that correspond to finite left cosets of the cyclic group generated by $g$).