Counterexample: $X=\{0,1\}^{\mathbb{N}}$ is the space of sequences of zeros and ones, $\mathcal{A}$ is the product $\sigma$-algebra, $\mu$ is the direct product of measures on $\{0,1\}$ that assign $\frac{1}{2}$ to $\{0\}$ and $\frac{1}{2}$ to $\{1\}$, $\phi$ is the shift mapping that sends each $(x_0, x_1, x_2, ...)$$(x_0, x_1, x_2, \dotsc)$ to $(x_1, x_2, x_3, ...)$$(x_1, x_2, x_3, \dotsc)$, and $A$ is the set $\{(x_0, x_1, x_2, ...) \mid x_0 = 0\}$$\{(x_0, x_1, x_2, \dotsc) \mid x_0 = 0\}$.
The question how much is lost by passing to $\phi^{-1}(\mathcal{A})$ is interesting. If nothing is lost, then by the same argument nothing is lost by passing to $\phi^{-2}(\mathcal{A})$ or $\phi^{-3}(\mathcal{A})$ etc. One would have to be more precise about the meaning of "dynamical information" ---— possibly the limiting behavior at infinity.