I will assume $(X,A)$ is a "good pair", i.e. that $A$ is a CW-subcomplex of $X$. Cup product works on mixed relative cohomologies, $H^p(X,A_1)\times H^q(X,A_2)\to H^{p+q}(X,A_1\cup A_2)$, so we can take $A_1=A_2=A$ or $A_1=A_2=\varnothing$ or $A_1=A$ and $A_2=\varnothing$ and vice versa. The coboundary map $H^n(A)\to H^{n+1}(X,A)$ is obtained by taking co-chains on $A$ and viewing them as co-chains on $X$ which vanish on $X-A$ and then pre-composing with the differential $C_{n+1}(X)\to C_n(X)$, i.e. it is obtained directly from the co-differential. If you work out the formula for the co-differential of the cup product of co-chains (which is the Leibniz rule), this should respect the values of the relative co-chains... but now I'm stuck, and the best I can say is the following:
If $\beta=i^\ast\eta$ where $i:A\hookrightarrow X$ and $\eta\in H^\ast(X)$, then the desired formula is $$\partial_{p+q}(\alpha\smile i^\ast\eta)=\partial_p\alpha\smile\eta$$
This is a "stability" result found in chapter VII section 8 of Dold's Lectures on Algebraic Topology. Exercise #3 of that section asks for a generalization of this result, which mimics the corresponding result for cross products (given in section 7 and section 2 of the same chapter). But while the cross product makes sense for general pairs $(X,A)$ and $(Y,B)$, the cup product needs $X=Y$ so that we may apply the diagonal map $\Delta:X\to X\times X$ (and appropriate relative versions). So I think Dold's desired "generalization of stability" only considers larger spaces such as $(X\times Y,A\times Y)$.
I originally wrote down a "Leibniz rule", but it's not defined (see the comments). Though for the cross product it is the case that $\partial_{p+q}(\alpha\times\beta)=\partial_p\alpha\times\beta=\pm\alpha\times\partial_q\beta$$\partial_{p+q}(\alpha\times\beta)=\partial_p\alpha\times\beta=(-1)^p\alpha\times\partial_q\beta$.