I've never seen an authoritative explanation for the choice of the lower case letter $\ell$ or $l$ to denote an arbitrary prime different from a given prime $p$. This now has its own LaTeX command \ell, but has been in use at least since the old work of Taniyama and Weil involving *L* functions. That use of the upper case letter might have suggested the lower case here, I guess(?) The letter *q* would seem more natural in elementary number theory. The write-up of Serre's 1967 McGill lectures was published in 1968 by W.A. Benjamin under the title *Abelian l-adic representations and elliptic curves*. There his convention is to denote prime numbers by $\ell, \ell', p, \dots$, stating: "we mostly use the letter $\ell$ for $\ell$-adic representations and the letter $p$ for the residue characteristic of some valuation".

I've heard this question raised but not answered quite a few times. For instance, after a colloquium talk in Hamburg given by Bhama Srinivasan on Deligne-Lusztig characters, the elderly Ernst Witt asked the non-technical question I've just raised. (He had done impressive work in his youth but became a convert to the Nazi cause without apparently committing any war crimes. Possibly he was the young man reported to have shown up once at Emmy Noether's seminar wearing a pro-Nazi uniform. In old age he had retained some mental acuity but developed phobias about for example the flooring material in the math tower, which required talks like the ones Bhama and I gave to move to a remote building.)

[ADDED] Both Franz and quim point in the direction of how the symbol $l$ became common for prime numbers in Hilbert's development of Kummer's work. There he considers an $l$th root of unity ($l$ an odd prime) instead of $\lambda$ used earlier by Kummer. Later on I guess it became a default option for many people to use $l$ for a prime different from a given prime $p$, especially when $q$ became used commonly for a power of $p$.