Measuring the independence between the components of a stochastic process

Question

In a context of blind source separation (e.g. you want to extract the voice of a singer from a song), many approaches consist in maximizing the independence between the components of a certain decomposition of an input signal. However, the signals are often considered only as random variables, although their temporal structure is often strong. I am interested in reading / learning / developing about an appropriate generalization of this approach where the signal is now modeled as a stochastic process (and even a diffusion process is it makes things simpler). So first I need a manageable definition and measure of the independence between the components of a stochastic process.

For a random variable $X$ on $\mathbb{R}^n$, the Kullback-Leibler divergence between the joint and the product of the marginal distributions of $X$ is often used to measure the degree of independence between the components of $X$. It is written $$ I = \int p(X) log \frac{p(X)}{\prod_{a=1}^n p_a(X_a)}dX $$ where $p_a(X_a)$ is a marginal probability function. If $I=0$, the components are completely independent.

How can I generalize it to stochastic processes? On this link the definition of the independence between two stochastic processes is given. I guess this could lead to a formal generalization but I do not manage to write it. Can anyone do it?

Besides, I am afraid it would lead to unpractical computations. So I am thinking that maybe with sufficient strong conditions on the stochastic process (which will not be worse than the usual ergodic random variable approach), one could simplify this expression up to a practical one. For instance, assuming that the process is solution to a stochastic differential equation, I am dreaming of the power of Girsanov theorem to compute the fraction in the above equation...

Do you have the answers to my questions? Do you know any litterature on this topic? Any advice about the direction I should take?

Thanks for your help.

Answer 1

This has been addressed to some extent in the machine learning literature, though I'm not sure how helpful it will be for your problem. One way of defining the KL divergence between two distributions P and Q is the expectation (under P) of the log of the Radon-Nikodym derivative of Q with respect to P.

This definition works nicely for diffusion processes. The Radon-Nikodym derivative is known from Girsanov's theorem, so everything can be computed as long as the two diffusions of interest have the same (constant) diffusion coefficient.

The work of Archambeau et al covers a lot of this stuff: http://www.cs.ucl.ac.uk/staff/c.archambeau/publ/nips_ca07.pdf

Answer 2

For the Kullback-Leibler divergence to be finite you need (at the very least) the joint distribution of the components to be absolutely continuous with respect to the product. For stochastic processes this seems to be a very strong condition: for example, for continuous semimartingales this implies zero quadratic covariation, and indeed much more.