Yes.
Suppose $p$ were not continuous. Then we could find a sequence of signed measures $\mu_n$ with norms $\|\mu_n\| \le 2^{-n}$ but $|p(\mu_n)| \ge 1$. Let $|\mu_n|$ denote the total variation measure of $\mu_n$, which has the same norm as $\mu_n$. Set $\mu = \sum_n |\mu_n|$; this sum converges in the Banach space $C(X)^*$. Now each $\mu_n$ is absolutely continuous with respect to $\mu$ so there exists $f_n \in L^1(\mu)$ with $d\mu_n = f_n \,d\mu$, or in your notation, $\mu_n = \Phi_\mu(f_n)$. Moreover, we have $\|f_n\|_{L^1(\mu)} = \|\mu_n\|$, so $f_n \to 0$ in $L^1(\mu)$. Yet $|p(\mu_n)| = |p(\Phi_\mu(f_n))| \ge 1$, contradicting the continuity of $p \circ \Phi_\mu$ on $L^1(\mu)$.