Q1. My other answer shows that if we want $\ast$ to be associative, we'd better do something like restrict to abelian groups. So let's do that.

In that context, there is indeed a good analogue of the Möbius function. It's the function $\mu$ given by $$\mu(A) = \sum_{k = 0}^\infty (-1)^k c_k(A)$$ (for an abelian group $A$), where $c_k(A)$ is the number of chains $$1 = A_0 < A_1 < \cdots < A_k = A$$ of proper subgroups.

Why is this a good analogue? Well, write $\zeta$ for the function with constant value 1. The crucial property of the Möbius function is that it's the inverse, with respect to $\ast$, of $\zeta$: $$\mu = \zeta^{-1}.$$ And this is easily verified by a telescoping sum argument.

Incidentally, I guessed this formula for $\mu$ because something extremely similar is true for Möbius inversion for posets, and more generally categories. I haven't figured out yet whether this is an instance of general results about Möbius inversion for categories.

Edit: This construction appears to be due to Philip Hall: see Mike Spivey's answer.

1

Q1. My other answer shows that if we want $\ast$ to be associative, we'd better do something like restrict to abelian groups. So let's do that.

In that context, there is indeed a good analogue of the Möbius function. It's the function $\mu$ given by $$\mu(A) = \sum_{k = 0}^\infty (-1)^k c_k(A)$$ (for an abelian group $A$), where $c_k(A)$ is the number of chains $$1 = A_0 < A_1 < \cdots < A_k = A$$ of proper subgroups.

Why is this a good analogue? Well, write $\zeta$ for the function with constant value 1. The crucial property of the Möbius function is that it's the inverse, with respect to $\ast$, of $\zeta$: $$\mu = \zeta^{-1}.$$ And this is easily verified by a telescoping sum argument.

Incidentally, I guessed this formula for $\mu$ because something extremely similar is true for Möbius inversion for posets, and more generally categories. I haven't figured out yet whether this is an instance of general results about Möbius inversion for categories.