We can explicitly keep track of both the running average and the running minimum average.

Let $f(k,m,r)$ be the probability that after $k$ variables, the minimum average so far is $m$, and the current running average is $r$ with $m<r$.

Let $g(k,m)$ be the probability that after $k$ variables, the minimum average so far is $m$, and this is also the running average so far.

I claim that for $k\ge2:$
\begin{align}
f(k,m,r) &= \frac{e^{-kr}(kr)^{k-1}}{r(k-2)!}1_{[m<r]}\\
g(k,m) &= \frac{e^{-km}(km)^{k-1}}{(k-1)!}
\end{align}

Once we have these formulas, we can guess the limiting distribution from the fact that we are only interested in $f$ and not $g$ (since after many draws, the minimum average has almost surely happened in the past), and only in $r=1$ (since after many draws, the running average is almost surely 1). So we can guess that the limiting distribution is a normalization of $f(k,m,1)$, which we can read off as $1_{[m<1]}$, and is the uniform distribution that was desired.

More formally it is enough to show that
$$\int_0^\infty f(k,m,r)dr + g(k,m) \to 1_{[m<r]} \text{ as }k \to \infty$$
which I have verified numerically. The first term is just $\Gamma[k-1,km]/(k-2)!$, so the proof of the limit is probably easy even though I haven't found it yet.

Returning to the claim, the formulas for $f$ and $g$ can be proved by an induction for $k'=k+1$:
\begin{align}
f(k',m,r)=
&\int_{x=m}^{k'r/k} f(k,m,x)k'e^{-k'r+kx}dx \\
&+ g(k,m)k'e^{-k'r+km}\\
g(k',m)=
&\int_{r=m}^{\infty}\int_{x=m}^{r} f(k,x,r)k'e^{-k'm+kr}dx\,dr \\
&+ \int_{x=m}^{\infty}g(k,x)k'e^{-k'm+kx}dx
\end{align}
The four terms on the right-hand sides of those equations are just what is needed to keep track of the four possibilities for $m<r$ or $m=r$ and $m_{old}<r_{old}$ or $m_{old}=r_{old}$.