My understanding is that formality of the DGA $Ext^\bullet(X\oplus Y,X\oplus Y)$ implies 1), but also formality of $Ext^\bullet(X,Y)$ and $Ext^\bullet(Y,X)$ as bimodules over $Ext^\bullet(X,X)$ and $Ext^\bullet(Y,Y)$, and that this is a much stronger condition.
For instance, let $(E,p)$ be an elliptic curve over a field, work in the abelian category of coherent sheaves, let $X=\mathcal{O}$ and let $Y=\mathcal{O}_p$ be the skyscraper at $p$.
Then $Ext^\bullet(X,X)$ and $Ext^\bullet(Y,Y)$ are both (intrinsically) formal, but $Ext^\bullet(X\oplus Y,X\oplus Y)$ knows the affine coordinate ring of $E\setminus\{ p\}$ for the cubic embedding into $\mathbb{P}^2$. That's because one can iteratively build $\mathcal{O}(np)$ for $n>0$ as a twisted complex in $X$ and $Y$ (namely, $\mathcal{O}((n+1)p)$ is the twist of $\mathcal{O}(np)$ along the spherical object $Y$). Over an algebraically closed field, this gives a $j$-line of quasi-isomorphism classes of $A_\infty$-algebras $Ext^\bullet(X\oplus Y,X\oplus Y)$.
As requested a bit more detail on why 2) implies 1), probably by too clunky an argument.
Let $A=Ext^\bullet(X\oplus Y, X\oplus Y)$. We can regard this as an ordinary graded $K$-algebra, in which case non-formality of the $A_\infty$-structure is detected by the primary deformation class in $HH^\bullet_K(A,A)$. That is: after transferring the DG structure to a minimal $A_\infty$-structure on $A$ using homological perturbation theory, the composition $\mu^3$ defines a Hochschild cocycle. If it is a coboundary then we can kill $\mu^3$ by a gauge transformation which leaves $\mu^1$ and $\mu^2$ untouched, whereupon $\mu^4$ is a cocycle; and so on. If the structure is not formal, one will eventually obtain a non-trivial Hochschild class, called the primary deformation class.
We can alternatively regard $A$ as a 2-object graded-linear category, i.e., an algebra over $R=K\oplus K$, in which case non-formality is detected by a primary class in $HH^\bullet_R(A,A)$, defined similarly. But one checks using the bar resolution that $HH^\bullet_R(A,A)\cong HH^\bullet_K(A,A)$ as $K$-modules. Hence, if the algebra is formal, then so is the category; the restriction of the categorical primary deformation class to endomorphisms of $X$ is then trivial.
The references I tend to use for this sort of thing are the first chapter of Seidel's book "Fukaya categories and Picard-Lefschetz theory", and also his paper "Homological mirror symmetry for the quartic surface", but there are certainly other possibilities.
