Q1. My other answer shows that if we want $\ast$ to be associative, we'd better do something like restrict to abelian groups. So let's do that.
In that context, there is indeed a good analogue of the Möbius function. It's the function $\mu$ given by $$ \mu(A) = \sum_{k = 0}^\infty (-1)^k c_k(A) $$ (for an abelian group $A$), where $c_k(A)$ is the number of chains $$ 1 = A_0 < A_1 < \cdots < A_k = A $$ of proper subgroups.
Why is this a good analogue? Well, write $\zeta$ for the function with constant value 1. The crucial property of the Möbius function is that it's the inverse, with respect to $\ast$, of $\zeta$: $$ \mu = \zeta^{-1}. $$ And this is easily verified by a telescoping sum argument.
Incidentally, I guessed this formula for $\mu$ because something extremely similar is true for Möbius inversion for posets, and more generally categories. I haven't figured out yet whether this is an instance of general results about Möbius inversion for categories.
Edit: This construction appears to be due to Philip Hall: see Mike Spivey's answer.
