Here's a very general form. Suppose that we have six chain complexes $A_0, A_1, A_2, A_{01}, A_{12}, A_{012}$, with bilinear "multiplication" pairings of chain complexes:
$$
\begin{align*}
A_0 \otimes A_1 &\to A_{01}\\
A_1 \otimes A_2 &\to A_{12}\\
A_{01} \otimes A_2 &\to A_{012}\\
A_0 \otimes A_{12} &\to A_{012}
\end{align*}
$$
Let's assume that we also have a chain homotopy $H$ between the two composite maps $A_0 \otimes A_1 \otimes A_2 \rightrightarrows A_{012}$. Given cycles $x \in A_0, y \in A_1, z \in A_2$ of degrees $p$, $q$, and $r$ respectively such that $[x \cdot y] = 0 \in H_{p+q}(A_{01})$ and $[y \cdot z] = 0 \in H_{q+r}(A_{12})$, then we can form a Massey product: choose elements $u$ and $v$ such that $\partial u = x \cdot y$ and $\partial v = y \cdot z$, and form
$$
\langle x,y,z\rangle = u \cdot z - (-1)^p x \cdot v + H(x\otimes y \otimes z)
$$
which represents an element in $H_{p+q+r+1}(A_{012})$. This depends in the usual way on the choices of $u$ and $v$, so there is indeterminacy. (If you use cohomological indexing then for elements in cohomological degrees $p$, $q$, and $r$ you get a Massey product in $H^{p+q+r-1}(A_{012})$.)
The "usual" definition of a Massey product is when all six chain complexes are equal to a single complex $A$ equipped with a strictly associative multiplication (allowing us to choose $H = 0$) that makes it into a differential graded algebra.
In the case you describe, we can take
- $A_0 = A_{01} = C_*(X,pt)$ the relative singular chain complex of $X$,
- $A_2 = A_{12} = C_*(Y,pt)$ the relative singular chain complex of $Y$,
- $A_{012} = C_*(X \wedge Y, pt)$ the relative singular chain complex of $X \wedge Y$, and
- $A_1 = \Bbb Z$, the constant chain complex $\Bbb Z$ in degree 0.
Then two of the multiplications are just the unit isomorphism and the other two multiplications $C_*(X,*) \otimes C_*(Y,*)$ are induced by the Eilenberg-Zilber shuffle map. The unit isomorphism is associative and so we can take $H = 0$ in this case, recovering your formula.
If you like, the entire framework of these six objects, pairings, homotopy, and chosen elements asks for a DG-category (really an $A_\infty$ DG-category) with four objects and chosen maps
$$
a \stackrel{z}{\to} b \stackrel{y}{\to} c \stackrel{x}{\to} d
$$
so that the double composites are null; in this description we can also think of this secondary operation $\langle x,y,z\rangle$ as related to Toda's "bracket" construction. The case of a differential graded algebra is the one-object DG-category case.