Motivation or intuition is usually difficult to recover from older published work, but I think it helps a lot here to put the work of I.M. Gelfand and his collaborators in perspective. Finite groups of Lie type were never their main focus, and in any case by the 1970s the ideas of Deligne and Lusztig (developed much further by Lusztig and others) largely took over that subject. Gelfand looked much more broadly at representations of Lie groups and at linear algebraic groups over many kinds of local fields, along with finite fields.
For Gelfand's own work it's useful to look at the English translations of various papers in volume 2 of the Collected Papers (Springer, 1988), especially those gathered in Part IV: Models of representations; representations of groups over various fields. This includes the short 1962 papers with Graev which introduced Gelfand-Graev characters. (Note that the Russian papers are now freely available online, but usually it's harder to get access to the translation journals -- and some of the translations are less reliable than others.)
It's worth quoting the opening lines of a 1974 note by Bernstein-Gelfand-Gelfand A new model for representations of finite semisimple algebraic groups (in a sometimes non-idiomatic English translation):
"An important problem in representation theory is to construct the so-called model, i.e. representations of the given group $G$ that contain almost every irreducible representation of $G$ exactly once."
This theme was pursued for various classes of Lie groups (compact and noncompact) and finite groups of Lie type, where the technical methods naturally vary a lot. But roughly speaking the emphasis is placed on working with induced representations from a maximal unipotent subgroup. In the finite case, this presumably led to experimentation with inducing the simplest types of characters (those of degree 1) from a Sylow $p$-subgroup such as the upper triangular unipotent subgroup in a finite general linear group. Since the subgroup is relatively small in this case, it's of course not to be expected that such an induced character is irreducible. Indeed, if one induces the trivial character of a Borel subgroup such as the full triangular subgroup of a general linear group, the decomposition of the induced character is complicated to work out. But if the ambient algebraic group has a connected center, the more subtle idea of Gelfand-Graev is remarkably successful: inducing a "regular" (meaning most non-trivial) character of the unipotent group yields an induced character whose constituents all have multiplicity 1 and exhaust the "regular" characters of $G$. Connectedness of the center is essential for getting a unique character of $G$ regardless of which regular character of the unipotent group is used in the construction.
(Gelfand-Graev could only prove the multiplicity 1 property in a special case, but Steinberg then provided a general proof given in Carter's book. It's worth looking at Steinberg's discussion toward the end of his last section 14; his 1967-68 Yale lectures are still available online here.)
This notion of "regular" character was later shown by Lusztig to fit well with the Deligne-Lusztig theory, in terms of a more precise analogue of the notion of "regular" element in an algebraic group. See also Chapter 14 in the concise textbook by Digne and Michel Representations of Finite Groups of Lie Type (Cambridge, 1991).
