I will assume $(X,A)$ is a "good pair", i.e. that $A$ is a CW-subcomplex of $X$. Cup product works on mixed relative cohomologies, $H^p(X,A_1)\times H^q(X,A_2)\to H^{p+q}(X,A_1\cup A_2)$, so we can take $A_1=A_2=A$ or $A_1=A_2=\varnothing$ or $A_1=A$ and $A_2=\varnothing$ and vice versa. The coboundary map $\delta: H^n(A)\to H^{n+1}(X,A)$ is obtained by taking co-chains on $A$ and viewing them as co-chains on $X$ which vanish on $X-A$ and then pre-composing with the differential $\partial:C_{n+1}(X)\to C_n(X)$, i.e. it is obtained directly from the co-differential. If you work out the formula for the co-differential of the cup product of co-chains (which is the Leibniz rule), this should respect the values of the relative co-chains... but now I'm stuck, and the best I can say is the following:

If $\beta=i^\ast\eta$ where $i:A\hookrightarrow X$ and $\eta\in H^\ast(X)$, then there is a "stability" property
$$\delta(\alpha\smile i^\ast\eta)=\delta\alpha\smile\eta$$

This can be found in chapter VII section 8 of Dold's *Lectures on Algebraic Topology*. Exercise #3 of that section asks for a generalization of this result, which mimics the corresponding result for cross products (given in section 7 and section 2 of the same chapter). Now while the cross product makes sense for general pairs $(X,A)$ and $(Y,B)$, the cup product needs $X=Y$ so that we may apply the diagonal map $\Delta:X\to X\times X$ (and appropriate relative versions). So I think Dold's desired "generalization of stability" is only considering larger spaces such as $(X\times Y,A\times Y)$.