Addendum I started this project with the following problem. Let $\mathfrak{g}$ be a simple Lie algebra whose dimension is $D$. Normalise the Casimir so it acts as 1 on $\mathfrak{g}$. Now consider the subspace of the exterior power $\wedge^k \mathfrak{g}$on which the Casimir acts by $k$. This is a representation of $\mathfrak{g}$ but obviously does not make sense for $k>D$. Taking $\mathfrak{g}=\mathfrak{sl}(5)$ we have $\tau(k)$is the dimension of a representation for small $k$ (certainly no more than 24). I doubt this is interesting.

The conclusion of the project was that for all $k$ there is a complex of representations of $\mathfrak{g}$. Then the Euler characteristic is a virtual representation. This can be written as a sum (with signs) of representations of $\mathfrak{g}$ using the MacDonald identities for affine $\mathfrak{g}$. This gives $\tau(k)$ as the dimension of a virtual representation of $\mathfrak{sl}(5)$ for all $k$.

Because of the signs this does not give an immediate solution to Lehmer's question. However it is a different way of looking at the problem.

I also give an explicit formula for $\tau(k)$ in terms of partitions and hooklengths.I believe this is new.

1

I don't know if this helps but you can put $D=24$ in (13) (14) of my paper to get an explicit formula for $\tau(n)$.

MR2218820 (2007c:17009) Westbury, Bruce W. Universal characters from the Macdonald identities. Adv. Math. 202 (2006), no. 1, 50--63.

doi:10.1016/j.aim.2005.03.013

Since $SL(5)$ is a simple Lie algebra of dimension 24 this also relates $\tau(n)$ to the affine root system of type $A_4$.

I doubt Lehmer would have had this in mind.