Let $H$ be a finite group, representation theory of $H$ over $\Bbb C$ essentially determines $\operatorname{Hom}(H,GL_n(\Bbb C))$ up to conjugation action of $GL_n(\Bbb C)$ for each $n$. If we replace $GL_n(\Bbb C)$ by $G(\Bbb C)$ where $G$ is a linear algebraic group, what about the classification of $\operatorname{Hom}(H,G(\Bbb C))$ up to conjugation? For any $h \in \operatorname{Hom}(H,G(\Bbb C))$, what is the stabilizer of $h$ under conjugation(In the case $G=GL(V)$ this recovers the notion of $\text{Aut}_G(V)$)? Could we develop other similar notations in representation theory of finite groups?
Maybe it's better to consider a system of linear algebraic groups $\{G_i\}$ with closed immersions $G_i \times G_j \hookrightarrow G_{i+j}$. For example, the projective representation theory can be developed such as the notion of characters (See here). One difficulty is that if two elements are conjugated in a larger group, they may not be conjugated in the small group.
Another motivation is when considering representations of groups, sometimes we come across other algebraic groups like $PGL_n$ such as in the construction of Weil representations.
Currently I only have some vague ideas about this problem. I am mostly interested in the case $G=GSp_n$, $G=SL_n$, $G=B_n$ and $G=U(n)$. (Of course $U(n)$ is not an algebraic group over $\Bbb C$ and we consider $\{ A \in M_n(\Bbb C)| AA^*=I \}$).
Edit: Thanks for pointing out some good references. I now still have two special questions: is $\operatorname{Hom}(H,G(\Bbb C))$ finite and equal to $\operatorname{Hom}(H,G(\bar{\Bbb Q}))$ (both up to conjugation)? If $G$ is abelian, both of them are true as we can reduce to the case $H$ is connected and reductive as nilpotent radical is torsion free and then reduce to the toric case as all maximal torus are conjugated. Also, if $H=SL_n,GL_n,PGL_n$ the result is true by (projective) representation theory (the Schur multiplier $H^2(G,\Bbb C^{\times})=H^2(G,\bar {\Bbb Q}^{\times})$ is finite). My vague idea to attack those two special problems is to regard $\operatorname{Hom}(G,H(F))$ as $F$-points of an algebraic variety for any field $F$ with action by $H$, and maybe the theory of actions of linear algebraic group gives the answer.
