As Benjamin Steinberg point out in his answer, this result is proved (probably more efficiently) in an Ohio State University undergraduate honors thesis (2012) by
Paul Apisa
We claim in the finite case that this can only happen for $p$-groups, where $p$ is a prime : Suppose first that the finite group $G$ has a unique up to automorphism irredundant generating set of minimal cardinality, say $\{g_{1}, g_{2}, \ldots g_{n} \}.$ - this condition seems a priori weaker than that asked in the question. I claim that each $g_{i}$ has the same order. For if $g_{1}$ and $g_{2}$ have different orders, then $\{g_{1}, g_{1}g_{2}, g_{3},\ldots g_{n} \}$ and $\{g_{1}g_{2}, g_{2},g_{3}, \ldots g_{n} \}$ are also irredundant generating set of the same minimal cardinality, and the sum of the orders of the given generators is fixed by hypothesis, yielding a contradiction. Also, for each $i,$ we see that $\langle g_{i} \rangle$ must be maximal cyclic subgroup of $G,$ for if $\langle g_{i} \rangle$ is strictly contained in $\langle h_{i} \rangle , $ then
$\{g_{1}, g_{2}, \ldots, g_{i-1}, h_{i},g_{i+1}, \ldots g_{n} \} $ is an irredundant generating set of the same minimal cardinality, but the sum of the order of its elements is larger than before.
But now I claim in the context of the original question (in which we work from now on) that we may assume that each $g_{i}$ has (the same) prime power order. For if we could write $g_{1} = a_{1}b_{1} = b_{1}a_{1}$ with $a_{1},b_{1}$ of coprime orders, neither of which is $1,$ then by what we have already established,
$\{a_{1},b_{1},g_{2},g_{3}, \ldots, g_{n} \}$ is an irredundant generating set for $G$ which clearly can't be obtained by applying an automorphism of $G$ to $\{g_{1},g_{2},\ldots, g_{n} \}.$ Now each $g_{i}$ has order $p^{k}$ for some fixed prime $p$ and positive (assuming $G$ is non-trivial!) integer $k$.
Now every maximal subgroup of $G$ requires at least $n-1$ generators, by the minimality of $n.$ Furthermore, if there is a maximal subgroup $M$ of $G$ which requires $n-1$ generators, then every element of $G \backslash M$ has order $p^{k}.$ It follows in that case that $O^{p}(G) \leq M,$ and that $[G:M] = p$ (and $M \lhd G ).$ Now if every maximal subgroup of $G$ is normal, then $G$ is nilpotent, and hence is a $p$-group.
Hence we may suppose that $G$ has a maximal subgroup $H$ which can be generated by
$m \geq n, $ but no fewer, elements. Also, we may suppose that there is an element $y \in G \backslash H$ which does not have order $p^{k}.$ Let $\{h_{1},h_{2},\ldots h_{m} \}$ be an irredundant generating set for $H.$ We claim that $\{h_{1},h_{2},\ldots h_{m},y \} $ is an irredundant generating set for $G,$ which contradicts the transitivity of ${\rm Aut}(G)$ on irredundant generating sets. We certainly can't omit $y$ from the generating set. If we could omit one or more $h_{i},$ then we would have a minimal irredundant generating set obtaining an element not of order $p^{k},$ contrary to what we have already established. But now we have an irredundant generating set with $m+1 > n$ elements, still in contradiction to the transitivity assumption.
Hence any finite group $G$ with this transitivity of ${\rm Aut}(G)$ on irredundant generating sets is a $p$-group for some prime $p.$