Signal model classification between two possbile candidates

Question

How to decide the most possible signal model between two model candidates besed on the received signal vector?

Assume the received signal vector is $y$, the possible signal model candidates could be:

(1) $y = Ax+n$, or

(2) $y = Bx+n$,

in which $x$ is the transmitted signal vector, and $A$ and $B$ are the system matrices for signal model candidate-1 and candidate-2 respectively, and $n$ is the Gaussian noise vector.

If $y$,$A$ and $B$ are all known, and the noise covariance matrix is $E[nn^H] = w^2I$, in which $w^2$ stands for noise power, and $I$ is the identity matrix, how to decide the most possible signal model candidate between the two. What's the optimal solution?

Thanks for any discussions.

Answer 1

A good reference book could be

Hero, A. “Signal Detection and Classification” Digital Signal Processing Handbook Ed. Vijay K. Madisetti and Douglas B. Williams Boca Raton: CRC Press LLC, 1999

in which different scenarios were discussed. Besides, some reference books are also helpful.