The answer here is certainly yes under many sets of mild side hypotheses.
For example, once upon a time, I wrote a paper with Frank Adams (!) that seems of some relevance: [J.F.Adams and N.J.Kuhn, Atomic spaces and spectra, Proc. Edin. Math. Soc 32 (1989), 473-481]. We show that if $X$ is a space or spectrum that is $p$--complete and of finite type, then the monoid of homotopy classes of based self maps $[X,X]$ is a profinite monoid. In this case, one concludes that, if $X$ has no nontrivial retracts then every self map is either invertible or topologically nilpotent.
Let's apply this to the stated question, under the hypotheses $A$ has no nontrivial retracts and is complete of finite type. Consideration of homology shows that at least one of the two maps $A \rightarrow X \rightarrow X_i \rightarrow X \rightarrow A$ is not topologically nilpotent, and so must be an equivalence. Thus $A$ is a retract of either $X_1$ or $X_2$, i.e. the question has an affirmative answer in this case.
More generally, you are asking something closely related to a Krull-Schmidt type theorem for spaces: if a space $X$ is written as a wedge of `indecomposable' spaces in two different ways, must the pieces correspond? Issues here include: need a space be written in this way? and also: What is the difference between retracts and wedge summands?
If $X_1 \vee X_2$ is a suspension, and suitably complete, then certainly a Krull-Schmidt theorem holds and the answer is yes. In the more algebraic world of spectra, some of us were using these ideas all the time in the early 1980's in papers about stable splittings of classifying spaces.