Zhaoliang,

Maybe you wanted to ask this question:

Let $A$ and $B$ be two $n \times n$ real symmetric matrices such that
$$ \det(I_n-xA)\det(I_n-yB) = \det(I_n - xA-yB)$$
holds for all real values of $x$ and $y$.
Then $A B = 0$.

There are many proof, my favorite is probably a short proof in the paper *On a matrix theorem of A. T. Craig and H. Hotelling* by Olga Taussky.

You can also assume only that $\forall x\in \mathbb{R}, \det(I_n-xA)\det(I_n-xB) = \det(I_n - xA-xB)$, then you still have $AB=0$, but this is not in Taussky's article.

For those of you interested, here is a variant:
If $\mathcal{S}\subset \mathbb{R}$ such that $|\mathcal{S}|=n^2$, and $\forall x\in \mathcal{S}, \det(I_n-xA)\det(I_n-xB) = \det(I_n - xA-xB)$, do we necessarily have $AB=0$?

`$X\cap Y=\emptyset$`

repair the problem? – Harald Hanche-Olsen Mar 30 '10 at 15:57`$X\cap Y=\emptyset$`

and`$AB=0$`

then one of the two matrices is invertible, hence the other is the zero matrix. Hence, if this was the formulation of the problem then the conclusion holds in a trivial sense, which at least makes the problem not so good. I could change my extra condition into`$X\cap Y\subseteq\{0\}$`

, but I'd better stop my idle guessing game now. – Harald Hanche-Olsen Mar 30 '10 at 17:57