Zhaoliang,
Maybe you wanted to ask this question:
Let $A$ and $B$ be two $n \times n$ real symmetric matrices such that
$$ \det(I_n-xA)\det(I_n-yB) = \det(I_n - xA-yB)$$
holds for all real values of $x$ and $y$.
Then $A B = 0$.
There are many proof, my favorite is probably a short proof in the paper On a matrix theorem of A. T. Craig and H. Hotelling by Olga Taussky.
You can also assume only that $\forall x\in \mathbb{R}, \det(I_n-xA)\det(I_n-xB) = \det(I_n - xA-xB)$, then you still have $AB=0$, but this is not in Taussky's article.
For those of you interested, here is a variant:
If $\mathcal{S}\subset \mathbb{R}$ such that $|\mathcal{S}|=n^2$, and $\forall x\in \mathcal{S}, \det(I_n-xA)\det(I_n-xB) = \det(I_n - xA-xB)$, do we necessarily have $AB=0$?
$X\cap Y=\emptyset$
repair the problem? $\endgroup$$X\cap Y=\emptyset$
and$AB=0$
then one of the two matrices is invertible, hence the other is the zero matrix. Hence, if this was the formulation of the problem then the conclusion holds in a trivial sense, which at least makes the problem not so good. I could change my extra condition into$X\cap Y\subseteq\{0\}$
, but I'd better stop my idle guessing game now. $\endgroup$