**revised**

It is not always (perhaps not even "usually") the case that "most" pairs of vertices are "much" less than the diameter apart. Based on meager experience (see below), it seems common that "most" pairs are "almost" the diameter apart, but relatively few are exactly that far apart. Asking that the average distance (ad) or square root of the average squared distance (rasd) are very close to the diameter (either with difference less than 1 or ratio going to 1$ is a much stronger condition.

A small experiment I tried (for degree 3 or 4) was: choose two permutations of $t$ objects, then look at the group and Cayley graph determined by these two (and their inverses.) For my choices (all with $t \le 9$) the group was usually $A_t$ or $S_t.$ The ratio of the ad and rasd to the diameter seemed (in the best cases) to increase with $t.$ Three examples:

The permutations $(1 2)(3 4) (5 6)$ and $(1 7 8 5 3 4 6 9 2)$ generate $S_9$ creating a graph of degree $3$ with $9!=362880$ vertices and diameter $22.$ The distribution of distances is $$3, 6, 12, 24, 46, 90, 176, 344, 672, 1310, 2531, 4867, 9270, 17201, 30867, 51354$$ $$ 75493, 86173, 61359, 19347, 1699, 35$$ leading to $17.03$ and $ 17.14$ for the ad and rasd (this was the best of 5 trials)

The permutations $(1 2 3 4 5 6 7 8)$ and $( 1 6 7 2 3 5 8 4)$ generate $S_8$ creating a graph of degree $4$ with $8!=40320$ vertices and diameter $13.$ The distribution of distances is $$4, 12, 34, 94, 250, 648, 1642, 3939, 8275, 12468, 9843, 2998, 112$$ leading to $9.76$ and $ 9.86$ for the ad and rasd (this was the best of 15 trials)

Two random 7-cycles will usually generate $A_7$ (2520 vertices, degree 4). Out of 100 trials, 6 came out with diameter $9$ and distance distribution $4, 12, 34, 92, 252, 573, 936, 582, 34$ giving ad and rasd of $6.6336$ and $ 6.7456.$ (and none were better.) Perhaps someone can figure out (rather than observe from random trials) what optimum choices are for two $t$-cycles or a $t$-cycle and an involution.

Following a suggestion by @Gerhard: A Rubik's cube (with unmarked center faces) is known to have $n=43,252,003,274,489,856,000$ positions. There are $18$ basic moves (counting a half turn as a single move). This defines a vertex transitive graph of degree 18. It was long suspected, and now is known, that this graph has diameter 20. The team which proved this released numbers showing approximately how many nodes are at distance $d$ from a given one. The numbers, (being approximate) do not add exactly to the correct value of $n$ although they come close. After $1,18$ each number is roughly $13$ times the previous until the last few where the ratios are about $12,2.5,0.05$ and finally $2\cdot10^{-10}$ Using the given numbers, the ad and rasd are both about $17.$

Here are some final comments on fixed degree: Fix a degree $k$ and suppose only that a graph is regular of degree $k$, $x$ be a particular vertex and all vertices are within $m$ of $x.$ The number at distance $d$ is at most $k(k-1)^{d-1}.$ It seems to clear that we want to maximize the number of vertices in the graph to have a relatively large average (squared) distance from $x$. If this count was exact for all $d \le m$ then the proportion at maximal distance would be close to $1-\frac1k.$ Then, for fixed $k$, the average distance and average squared distance would exceed $(1-\epsilon)m$ and $(1-\epsilon)m^2$ for large enough $m$. However, the diameter of the entire graph might be as large as $2m.$ Diameter of $m$ can only happen in a very few cases and not much is known about how close one can come to this in general. This paper about the degree diameter problem mentions a lower bound of $(\frac{k}{1.57})^m$ for the number of vertices. However I don't know the number of vertices at maximal distance.