Just ran into this old question. As other answers have noted (1) for fixed $n$, the number of groups of order $p^n$ depends on $p$ in a complicated way and (2) if you try to make an algebra with $p$-groups as a basis and multiplication by copying the Hall formula, it won't be commutative. I am writing to note that it won't be associative either!
In the Hall algebra, the coefficient of $[X]$ in $[A][B]$ is the number of submodules $Y$ of $X$ such that $Y \cong A$ and $X/Y \cong B$. If we make the analogous definition for groups, we would want the number of normal subgroups $Y \trianglelefteq X$ such that $Y \cong A$ and $X/Y \cong B$
So the coefficients of $[X]$ in ${\big(} [A][B] {\big)} [C]$ and in $[A] {\big(} [B][C] {\big)}$ are, respectively,
(1) The number of chains $Z \trianglelefteq Y \trianglelefteq X$ with $Z \cong A$, $Y/Z \cong B$ and $X/Y \cong C$ and
(2) The number of pairs $(Z, Y')$ where $Z \trianglelefteq X$ and $Y' \trianglelefteq X/Z$, with $Z \cong A$, $Y' \cong B$ and $(X/Z)/Y' \cong C$.
Replacing $Y'$ by its preimage $Y$ in $X$, (2) is the same as
(2') The number of chains $Z \trianglelefteq Y \trianglelefteq X$ which have $Z \trianglelefteq X$, with $Z \cong A$, $Y/Z \cong B$ and $X/Y \cong C$.
In other words, the difference between (1) and (2') is that, in (1), we require that $Z \subseteq Y \subseteq X$ is a subnormal series, and in (2') we require that it is a normal series.
Therefore, any $p$-group with a subnormal series that isn't a normal series will give a counterexample. For example, the dihedral group of order $8$ has $7$ subnormal series with quotients $(C_2, C_2, C_2)$, but only $3$ of them are normal series.