show/hide this revision's text 2 added 1552 characters in body

P.S. David's insightful perspective prompts me to add a few remarks about the fumbling I did in the early 1970s from a completely different direction based on inadequate algebraic methods. (At the time I knew little about Lie group representations and less about modern algebraic geometry.) For finite groups of Lie type, it was already a problem to understand the two large series of (mostly irreducible) characters of $SL_2(\mathbb{F}_q)$, having degrees $q+1, q-1$: "principal" and "discrete" series. The principal series is easily constructed by induction from a Borel subgroup, but the origin of the discrete series is mysterious.

After reduction mod $p$ of these characters (with $q$ a power of $p$), it was obvious that both series exhibited roughly the same behavior. Moreover, this paralleled closely the behavior of the "baby" Verma modules for the Lie algebra of the associated algebraic group (using for powers of $p$ the higher Frobenius kernels). In this example all such modules have dimension $q$, with highest weights of composition factors related by "linkage" under an affine Weyl group relative to $p$. I became convinced in general that what I termed "deformation" of linkage classes made all series of finite group characters look essentially the "same"; by 1980 Jantzen had made this formalism precise. But meanwhile Deligne-Lusztig had found a more profound intrinsic way to understand the finite group characters. For me the main moral is the incredible unity found throughout Lie theory.

show/hide this revision's text 1

This is an extended comment, certainly not an answer to the very broad and perhaps unanswerable multiple part question being asked here ("How did people find those varieties?"). A facetious response might be: "With great difficulty and considerable ingenuity." But the question is certainly interesting and might someday inspire a monograph by an expert mathematician deeply interested in the history of late 20th century mathematics. Whether this person will exist is an open question.

Concerning the 1976 Annals paper by Deligne and Lusztig, they provide a certain amount of background discussion in their second section along with a very brief statement: Our work has been inspired by results of Drinfeld, who proved that the discrete series representations of $SL_2(\mathbb{F}_q)$ occur in the cohomology of the affine curve $xy^q - x^q y =1$.

Whereas classical work by Frobenius, Schur, and others, realized the character tables of these groups using orthogonality relations and induction techniques, there was never an explicit realization of the irreducible representations typically of degree $q-1$ (now called the "discrete series" by analogy with the Lie group case). Lusztig made a first investigation of such representations of finite general linear groups in his 1974 Annals of Mathematics Studies paper, motivated in part by the landmark combinatorial 1955 paper by J.A. Green (briefly a Warwick colleague). Ian Macdonald had proposed a general version of the results in that paper for other finite groups of Lie type, which Lusztig made the first serious attack on.

Meanwhile there had been a strong tendency toward translating group actions on (often projective) varieties into linear representations of the groups on (often finite dimensional) cohomology spaces. So a lot of history was coming together by the 1970s to encourage the approach of Deligne and Lusztig. Those wanting to get at least some insight from Lusztig's viewpoint should take a close look at his own comments on his papers including 17, 18, 22 now posted on his MIT webpage here.