Q2: In response to Greg Kuperberg's complaint about answering questions in comments, I'll combine the semi-answer in the comment above with F. Ladisch's specific example and put it here as a proper answer. As it's only half my own work, I'll make it community wiki.
For arbitrary f, g and h, we have $$ ((f \ast g) \ast h)(G) = \sum f(K) g(N/K) h(G/N) $$ where the sum is over all $K \triangleleft N \triangleleft G$, and $$ (f \ast (g \ast h))(G) = \sum f(K) g(N/K) h(G/N) $$ where the sum is over all $K \triangleleft G$ and $N \triangleleft G$ with $K \subseteq N$.
Take f, g and h all to have constant value 1. Then $((f \ast g) \ast h)(G)$ is the number of chains $K \leq N \leq G$ with K normal in N and N normal in G, whereas $(f \ast (g \ast h))(G)$ is the number of chains $K \leq N \leq G$ with K normal in G and N normal in G. The former is greater than or equal to the latter, and in some cases strictly greater: taking F. Ladisch's example, we might have $K = C_2$, $N = C_2 \times C_2$, $G = A_4$. So no, $\ast$ is not associative.
On the other hand, $\ast$ is associative when restricted to abelian groups.
