Given a finite group G and a prime p dividing the order of the group, is it possible to say if there are nontrivial representations of G over C whose reduction mod p has the trivial representation in its semisimplification? Is it possible to count such complex rep's, or classify them in some ways?

$\begingroup$ For an arbitrary finite group $G$, I wouldn't be at all optimistic about finding so much information relative to an arbitrary prime $p$ dividing its order. Is there any special class of groups (and primes) you want to understand? For example, groups of Lie type for the defining characteristic $p$ can lead to very complicated situations; but some special cases are worked out. (Note too that your "semisimplification" translates into the decomposition matrix for a given finite group and prime.) $\endgroup$ – Jim Humphreys Feb 28 '17 at 17:44

$\begingroup$ For the moment I seem to have no more information on G. It is a certain Galois group, so I guess pretty arbitrary finite group. May be one should begin with just the first question: what forces there to be no nontrivial complex rep'n which has trivial rep'n in its reduction mod p? $\endgroup$ – Dipendra Prasad Feb 28 '17 at 18:05

$\begingroup$ What is the reduction modulo $p$ of a complex representation? (Oh, I guess you're assuming that it preserves an integral structure?) $\endgroup$ – LSpice Feb 28 '17 at 18:21

1$\begingroup$ @ L.Spice: Yes, the idea of reduction modulo $p$ starts with a suitable integral structure in an ordinary irreducible representation (say over a $p$adic field). It is a basic but subtle fact proved by Brauer that every choice leads to the same composition factor multiplicities, which allows one to define decomposition numbers unambiguously. $\endgroup$ – Jim Humphreys Feb 28 '17 at 18:30
As for the existence question there is an easy way to see this must be the case:
Consider the regular representation $k[G]$ there is a unique (up to a scalar) map from the trivial representation to this where the image is the space spanned by the sum of all elements of $G$.
Similarly there is a unique (up to scaling) map the other direction sending some formal linear combination of elements of $G$ to the sum of the coefficients.
If $p=char(k)$ divides $G$ then these maps compose to zero and hence neither splits. In particular $k[G]$ has at least two trivial composition factors.
But of course $k[G]$ is the reduction mod $p$ of the regular representation in characteristic zero which only has one copy of the trivial. Hence some other irreducible must contribute a trivial composition factor when reduced mod $p$.

$\begingroup$ Thank you Nate for this beautiful argument. It answers my initial question for which group there are nontrivial irreducible representations whose reduction mod p contains the trivial representation. You answer says `always' if p divides the order of the group. This is what I was hoping to $\endgroup$ – Dipendra Prasad Mar 1 '17 at 13:01

$\begingroup$ know. I know for PGL(2,F_p), there is a unique irreducible rep'n with this property (of dim = p1, a cuspidal representation) so your answer is best possible for the moment. $\endgroup$ – Dipendra Prasad Mar 1 '17 at 13:03
To expand my short comment, I think it's possible to say something at least qualitative about the existence question you've raised. This goes back to Brauer's early work, some done with his student Nesbitt. For a modern treatment, see for example Serre's textbook or the first volume of Curtis and Reiner's Methods of Representation Theory. Here the fields involved are big enough to be splitting fields. (But your further question is far more difficult to answer in general.)
A basic fact, for given $G$ and $p$, is that the decomposition matrix $D$ records the multiplicity of each modular irreducible as a composition factor of (any) reduction mod $p$ of an ordinary character. One convention is that the rows of $D$ are indexed by the ordinary characters and the columns indexed by the modular (or Brauer) characters. As Brauer showed, the number of rows $r$ is the number of conjugacy classes of $G$, while the number of columns $s$ is the number of conjugacy classes containing elements of order relatively prime to $p$.
Then a fundamental reciprocity discovered by BrauerNesbitt is given by the matrix equation $C = D' D$; here $C$ is the $s \times s$ matrix of Cartan invariants, giving the multiplicities of modular irreducibles as composition factors of their various projective covers (or PIMs), while $D'$ denotes the transpose of $D$. As a consequence, $C$ is symmetric. Moreover, every PIM has dimension divisible by the power of $p$ dividing $G$. Therefore the trivial modular character does occur once in its PIM, but it must also occur in the reduction mod $p$ of some other ordinary character.
On the the other hand, the projective cover of the trivial module is typically the most complicated one to pin down as to its precise dimension and as to which ordinary characters contribute to it. For finite groups of Lie type, the other extreme case is the Steinberg module, which lies in a block of its own and is its own projective cover. Its dimension is just the power of $p$ dividing $G$.
ADDED: I should also refer to my 1975 exposition here (which is premodern, meaning before DeligneLusztig) on ordinary and modular representations of SL$_2(\mathbb{𝔽}_p)$. This explains that the PIM for the trivial module has dimension just $p$, involving the trivial character of degree 1 and a second "cuspidal" character of degree $p1$; but in general this dimension gets far more complicated to compute and the PIM involves a greater variety of ordinary characters reduced mod $p$.

$\begingroup$ Thank you for a beautiful summary of some of the modular theory. Nate has answered what I was looking for since as I wrote in my response to him, his answer cannot be improved. $\endgroup$ – Dipendra Prasad Mar 1 '17 at 13:05