To reinforce Geoff's answer, I'd emphasize that the "fairly standard exercise" mentioned in the question is only standard when you consider characteristic 0 irreducible representations as is usually done in a first course (then it's also usual to start with a big field like $\mathbb{C}$ to avoid extra complications). When dealing with modular representations, there are lots of further issues.
Qiaochu attempts an outline of the "standard exercise" which shows clearly where problems arise. In his first step, you need characteristic 0 (and a splitting field) to be confident that the degree of an irreducible representation divides the group order. This is definitely false in general. And then of course characteristic 2 is especially dangerous for dealing with the negative of the $2 \times 2$ identity matrix.
The bottom line is that you have to avoid this "standard exercise" and focus just on the special situation you are studying, in order to avoid confusion.
[ADDED] 1) Concerning the "standard exercise" (say over $\mathbb{C}$), I recall that decades ago it appeared as a Math Monthly problem (which I wrote an answer for) and later became an exercise in at least one advanced algebra text. Given an absolutely irreducible representation of degree 2 for a (necessarily nonabelian) finite simple group, classical character theory implies that the group order is even, so you can apply Cauchy etc. Of course, Feit-Thompson gets you there as well, but that's overkill. However, in odd characteristic that does help to get through the other steps of the proof, but in characteristic 2 you'd still face $I = -I$.
- While the degrees of irreducible representations of simple groups of Lie type are far from being known explicitly in general, over a splitting field of the defining characteristic you can see easy examples where degrees fail to divide the group order. For instance, take $\mathrm{PSL}_2(\mathbb{F}_7)$ in characteristic 7. Here you have an irreducible of degree 5 whereas 5 doesn't divide the group order 168.