In representation theory of finite groups, there is Mackey test for irreducibility of an induced representation (Serre Linear Representations of Finite Groups  #7.4). The author has stated it for field $\mathbb{C}$. Is it true for representations over arbitrary field whose characteristic is zero or coprime to the order of group ?

$\begingroup$ A good reference for this seems to be D. Bump's Lie Groups (chapter 34 and further). Bump comments on the dependence on the ground field. $\endgroup$– Igor RivinDec 29, 2011 at 13:31
3 Answers
I think so. The key property used in the proof is $$ Res^G_H Ind^G_H V=\bigoplus_{s\in H\setminus G/H} Ind^H_{H_s}V_s, $$ where $V_s$ is the twist of $V$ by $s$, and $H_s=sHs^{1}\cap H$, and its proof is more or less manifestly characteristicfree. Other than that, instead of characters and scalar products $\langle \chi_U,\chi_W\rangle$ you can use the actual spaces of intertwiners $Hom_G(U,W)$, and finally instead of saying that irreducibility of $W$ means the intertwining number $\dim Hom_G(W,W)$ is equal to 1 (which is true over an algebraically closed field of characteristic zero), you can say that irreducibility of $W$ means that $Hom_G(W,W)$ is a division algebra (which is true in the semisimple case, so under the assumption you want to make).
Let me, following a comment of Darij Grinberg, make the last step a bit more clear. There is a natural homomorphism of algebras $Hom_H(V,V)\to Hom_G(Ind^G_HV, Ind^G_HV)$. The induced representation $Ind^G_HV$ is irreducible if and only if this map is an isomorphism of algebras. However, to conclude that it is sufficient to show that it is an isomorphism of vector spaces, which we do without trouble via the Frobenius reciprocity.

$\begingroup$ We get $\Hom_G\left(V,V\right) = \bigoplus\limits_{s\in H\backslash G / H} \Hom_H\left(\rho,\rho^s\right)$ as vector spaces. I am pretty sure this doesn't say anything about their ring structure, if the RHS has any at all; how do you see that $\Hom_G\left(V,V\right)$ is a division algebra iff $ \Hom_H\left(\rho,\rho^s\right)=0$ for all $s\neq e$? $\endgroup$ Dec 29, 2011 at 13:36

$\begingroup$ I added a bit of a clarification. It's a kinda cheap trick with applying forgetful functors which I tend to like a lot. $\endgroup$ Dec 29, 2011 at 13:56

2$\begingroup$ The isomorphism of vector spaces $Hom_H(V,V)\rightarrow Hom_G(Ind^G_H V, Ind^G_H V)$ will hold if $V$ is irreducible and $Res^H_{H\cap H_s} V$ and $Res^{H_s}_{H\cap H_s}V_s$ ($s\in G\setminus H$) do not share an irreducible component; what about converse? $\endgroup$– josephJan 11, 2012 at 4:23

2$\begingroup$ You are right, the converse is actually not clear at all. Probably without any assumptions on a field you get only an if, not iff, statement. $\endgroup$ Jan 11, 2012 at 9:20

2$\begingroup$ (after one more minute of thinking) The natural 4d irreducible representation of the quaternionic group is induced from the nontrivial irreducible representation of the normal subgroup $\{1,1\}$! $\endgroup$ Jan 11, 2012 at 9:23
To reinforce what Vladimir says, I'd replace his "I think so" by "Yes". The basic paper by Mackey in Amer. J. Math. 73 (1951) is retrievable online through JSTOR and might be worth looking at. His initial discussion of finite groups is done over arbitrary fields of characteristic 0, intended to generalize the work of Frobenius while providing background for the study of locally compact groups. It's only a convenience to assume that the ground field is $\mathbb{C}$ (done by Serre in part because his audience for the early lectures consisted of people with applied interests in character theory).
By now there are multiple textbook sources (including Serre, Bump, Isaacs, Dornhoff, Grove, etc.), each with different coverage and emphases. Probably the most comprehensive treatment of induction (along with tensor products and intertwining maps) is found in Part I, Section 10, of the large treatise Methods of Representation Theory by Curtis & Reiner, which goes beyond their pioneering 1962 book. At first they can work over arbitrary commutative rings, while discussing only the general module results; this flexibility is needed later to pass to J.A. Green's work on modular representations.
But in order to get an applicable irreduciblity criterion for induced characters in (10.25), they take a subfield of $\mathbb{C}$ which is a splitting field for the given finite group and all its subgroups. Even this kind of assumption can be weakened, since you only need a field of characteristic 0 containing sufficiently many roots of unity to ensure absolute irreducibility of the characters of the various irreducible representations involved.
If you really want to work with fields of prime characteristic, it's also a standard principle that the character theory works the same way provided only that the characteristic of the field doesn't divide the group order. To deal with absolute irreducibility of characters, you still need the splitting field assumption.

$\begingroup$ As you have pointed out, for splitting fields, the theorem is true; no doubt about it. But I am not sure, whether it holds for nonsplitting field. Is the answer "Exactly YES"? $\endgroup$– josephJan 11, 2012 at 4:26

3$\begingroup$ Virtually all of the classical results on characters for finite groups (Frobenius, Schur, Mackey, etc.) are developed over a splitting field for the group and sometimes for all its subgroups too when induction is involved. It's usually a subtle question to sort out further questions about irreducibility vs. absolute irreducibility. The field of complex numbers is convenient for exposition but of course much too large. In any case, it seems unrealistic to expect clean general statements for an arbitrary field (of characteristic not dividing the group order). $\endgroup$ Jan 11, 2012 at 16:04
The inclusion ${\rm End}_H (V)\longrightarrow {\rm End}_G ({\rm Ind}_H^G V)$ is much clearer at the level of Hecke algebras. If ${\mathcal H}(G,V)$ denotes the spherical Hecke algebra attached to $V$ (which is known to be isomorphic to $ {\rm End}_G ({\rm Ind}_H^G V)$), then ${\rm End}_H (V)$ naturally identifies with the subalgebra of functions in the Hecke algebra with support $H$ : to $\psi$ you attach the function with support $H$ given by $h\mapsto \pi (h)\circ \psi =\psi \circ \pi (h)$. Then one has just to observe that the hypothesis of Mackey's criterion exactly says that functions in ${\mathcal H}(G,V)$ have support in $H$ !
For all of this you need to assume that the order of $G$ is prime to the caracteristics of the field of coefficients.