As the title says.
In particular, I am interested in the story for a general reductive group $G$, say defined over $\mathbb{Q}$. I can imagine that mod-$\ell$ (algebraic) automorphic representation should correspond to conjugacy classes of homomorphisms from the motivic Galois group into $\widehat{G}(\overline{\mathbb{F}_\ell})$, but this does not seem like a very workable definition. Is there a more pragmatic approach?
This is in my mind a central open problem.
Here is an explicit example which I believe is still wide open. Serre's conjecture (the Khare-Wintenberger theorem) says that if I have a continuous odd irreducible $2$-dimensional mod $p$ representation of the absolute Galois group of the rationals then it should come from a mod $p$ modular form. We have a perfectly good definition of mod $p$ modular forms (sections of the usual line bundles on mod $p$ modular curves).
But what is the "even" analogue of Serre's conjecture?
One problem is that in characteristic zero (i.e., for $2$-dimensional continuous complex representations, which must have finite image) these guys are expected to come from algebraic Maass forms, which have, as far as anyone knows, no algebraic definition: they are genuinely analytic objects and it is a complete miracle that their Hecke eigenvalues are algebraic. So there's a big open problem: give an algebraic definition of an algebraic Maass form which doesn't use analytic functions on the upper half plane which are eigenvectors for the Laplacian with eigenvalue $1/4$. I have no idea how to do this and I don't think that anyone else has either. The problem is that the forms are not holomorphic (so you can't use GAGA and algebraic geometry) and they're not cohomological (so you can't use group cohomology either), and those are in some sense the only tricks we have (other than Langlands transfer, but I don't know any functorial transfer of these guys which (a) loses no information and (b) gives rise to a form which is cohomological or holomorphic). It might be an interesting PhD problem to check this out in fact. Blasius and Ramakrishnan once tried to go up to an im quad field and then to $\mathrm{Sp}_4$ over $\mathbf{Q}$ but the resulting form isn't holomorphic. One might define an algebraic automorphic rep to be "accessible" if it's cohomological or perhaps limit of discrete series (perhaps these are the ones for which there's a chance of proving they're arithmetic), and then try and find examples of algebraic auto reps which should never transfer to an "accessible" rep on any other group.
But even after that, there's another problem, which is that as far as I know one does not expect a continuous even irreducible mod p representation of this Galois group to lift to a de Rham representation in characteristic zero (and indeed perhaps Frank Calegari might be able to give explicit counterexamples to this, after his recent observation that oddness sometimes follows from other assumptions). So even if one could give an algebraic definition of a Maass form, one wouldn't have enough Maass forms---they would all (at least conjecturally) give rise to Galois representations with images all of whose Jordan-Hölder factors are cyclic or $\mathrm{A}_5$ (by the classification of finite subgroups of $\mathrm{PGL}_2(\mathbf{C}))$. So there's another non-trivial sticking point.
In summary---nice question but I don't think that mathematics has a good answer yet (unless you're willing to just cheat and say that an automorphic representation "is" a Galois representation with some properties).
If $G$ is a connected, linear, reductive group over $\mathbb Q$, let $q_0$ be the dimension so denoted in Borel--Wallach, i.e. $q_0 = (d - l_0)/2,$ where $d = \text{dim } G_{\infty} - \text{ dim } A_{\infty}K_{\infty}$ and $l_0 = \text{rank } G_{\infty} - \text{ rank } A_{\infty}K_{\infty},$ where $G_{\infty} = G(\mathbb R)$, $K_{\infty}$ is a maximal compact subgroup of $G_{\infty}$, and $A_{\infty} = A(\mathbb R)$ with $A$ being the maximal $\mathbb Q$-split torus in the centre of $G$. (So $q_0$ is the first interesting dimension of homology in the locally symmetric arithmetic quotients attached to $G$.)
Then the inductive limit of the cohomology groups $H^{q_0}(G(\mathbb Q)\backslash G(\mathbb A)/A_{\infty}K_{\infty}K_f,\mathbb F_{\ell})$, as $K_f$ runs over all compact open subgroups of $G(\mathbb A_f)$, is an admissible smooth representation of $G(\mathbb G_f)$, which is (I believe) a reasonable substitute for a space of mod $\ell$ automorphic forms.
In the case when $G$ satifies the conditions of Gross's "Algebraic modular forms" paper, $q_0 = 0$, the locally symmetric quotients are just finite sets, and this is precisely the space of mod $\ell$ algebraic modular forms.
This doesn't deal with Kevin's problems; this space will (conjecturally) only contain automorphic forms whose systems of Hecke eigenvalues correspond to odd Galois representations. Nevertheless, it is a space, and one can (and does) try to work with it.
