Representation theory of finite groups with additional structures

Let $H$ be a finite group, representation theory of $H$ over $\Bbb C$ essentially determines $\operatorname{Hom}(H,GL_n(\Bbb C))$ up to conjugation action of $GL_n(\Bbb C)$ for each $n$. If we replace $GL_n(\Bbb C)$ by $G(\Bbb C)$ where $G$ is a linear algebraic group, what about the classification of $\operatorname{Hom}(H,G(\Bbb C))$ up to conjugation? For any $h \in \operatorname{Hom}(H,G(\Bbb C))$, what is the stabilizer of $h$ under conjugation(In the case $G=GL(V)$ this recovers the notion of $\text{Aut}_G(V)$)? Could we develop other similar notations in representation theory of finite groups?

Maybe it's better to consider a system of linear algebraic groups $\{G_i\}$ with closed immersions $G_i \times G_j \hookrightarrow G_{i+j}$. For example, the projective representation theory can be developed such as the notion of characters (See here). One difficulty is that if two elements are conjugated in a larger group, they may not be conjugated in the small group.

Another motivation is when considering representations of groups, sometimes we come across other algebraic groups like $PGL_n$ such as in the construction of Weil representations.

Currently I only have some vague ideas about this problem. I am mostly interested in the case $G=GSp_n$, $G=SL_n$, $G=B_n$ and $G=U(n)$. (Of course $U(n)$ is not an algebraic group over $\Bbb C$ and we consider $\{ A \in M_n(\Bbb C)| AA^*=I \}$).

Edit: Thanks for pointing out some good references. I now still have two special questions: is $\operatorname{Hom}(H,G(\Bbb C))$ finite and equal to $\operatorname{Hom}(H,G(\bar{\Bbb Q}))$ (both up to conjugation)? If $G$ is abelian, both of them are true as we can reduce to the case $H$ is connected and reductive as nilpotent radical is torsion free and then reduce to the toric case as all maximal torus are conjugated. Also, if $H=SL_n,GL_n,PGL_n$ the result is true by (projective) representation theory (the Schur multiplier $H^2(G,\Bbb C^{\times})=H^2(G,\bar {\Bbb Q}^{\times})$ is finite). My vague idea to attack those two special problems is to regard $\operatorname{Hom}(G,H(F))$ as $F$-points of an algebraic variety for any field $F$ with action by $H$, and maybe the theory of actions of linear algebraic group gives the answer.

• Even for a finite group as simple as $A_5$ (no pun intended), the question of classifying “irreducible” homomorphisms into e.g. a complex Lie group $G$ can be extremely complicated. For instance, in 2002 Lusztig showed that there is a unique irreducible $E_8(\mathbb{C})$ representation of $A_5$. See arxiv.org/abs/math/0202111v1 May 2, 2018 at 18:05
• B. Kuelshammer is one person who has written about this topic. May 2, 2018 at 18:13
• Serre (and many others after him) have considered the meaning of "complete reducibility" in this kind of setting: see his Bourbaki report and if you have access to MathSciNet follow up the many citations: numdam.org/item/SB_2003-2004__46__195_0 (or at least note the ample reference list in his Bourbaki talk). May 3, 2018 at 13:57
• P.S. Serre's earlier Moursund Lectures at U. Oregon (in English) also touched on relevant issues: math.uoregon.edu/resources/serre May 3, 2018 at 14:02
• Thank all of you for giving good materials for this topic, which I didn't know before. It seems a difficult problem for full classification while general theories are developed. May 3, 2018 at 17:00