In my studies of applied probability I have recently met the following problem on which I need help:

We consider two binary random (column) vectors $ X,Y \in \{0,1\}^d $ where the mutual distribution satisfies $ P(X,Y) = \frac{e^{-X^TWY-b^TX-c^TY}}{Z(W,b,c)} $ where W is a symmetric positive semidefinite matrix, b and c are constant column vectors of length $ d $ and $ Z $ is the partition function to normalize the probability distribution, this setting might be familiar to some from machine learning and neural networks as a "Restricted Boltzmann Machine" (RBM), I was looking at the mutual information $ I(X;Y) = \sum_{X} \sum_{Y} P(X,Y) \ln{(\frac{P(X,Y)}{P(X)P(Y)})}$

I was wondering if we can approximate this quantity (the mutual information) or even calculate it analytically? We may assume the vectors are over {-1,1} if it makes things easier. Maybe an asymptotic approximation would help? I was thinking maybe a probabilistic approximation, maybe approximating the discrete distributions as Gaussian distributions or other distributions easier to deal with? I thank all helpers on this.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.