There is a paper P. Frankl and A. M. Odlyzko. "On subsets with cardinalities of intersections divisible by a fixed integer." European Journal of Combinatorics 4(3) (1983): 215–220, in which they provide an example of l-town on $(cl)^{[n/(4l)]}$ members for every (not nessesarily prime) integer $l$ and an ansolute constant $c$.
In the asymptotics this gives $2^{\frac{cn \ln l}{l}}$ which beats $2^{n/l}$.