Let $p$ be a prime such that the number of primes less than it is odd. Each number in the interval $[1-p,p]$ is a possible location for a sequence of $Z$s in the step directly after $p$ is added or subtracted. Some of these sequences end in $-1$, some do not. The convex hull of the numbers whose sequences end in $-1$ forms an interval. We will show, by induction, that this is about $[-p/2,p/2]$
Let $p_1$ and $p_2$ be the next two primes. If $x$ is the upper or lower bound for the interval of $p_2$, after applying the process twice it must be in the interval for $p_1$.
The upper bound: Clearly positive, so at the next step we subtract $p_1$. If this is still positive, then it is very close to $0$, so $u-p_1-p_2$ is very far from $0$ and not in the interval as long as the interval is approximately $[-p/2,p/2]$. So it's negative, and at the next step we add $p_2$. Thus the upper bound can increase by no more than $p_1-p_2$.
By identical logic, the lower bound can decrease by no more than the same amount. So it increases by a difference of two primes every two primes, so it's about half the size of the corresponding prime, so the hypothesis we need for induction is satisfied. it remains to check that this is true at some small prime. If my calculaions are correct for $p=41$ the interval is $[-19,17]$ which clearly qualifies, and we can manually check there are no solutions up till then.
Since the interval is small, the prime $p$ is never contained in it, and so the stated pattern occurs.