Computing the expectation of a quadratic matrix form involving Bernoulli and Gaussian distributed matrices

Question

I am working with two random matrices, $Z$ and $H$:

1. $Z$ is an $n \times K$ matrix with entries sampled i.i.d. from a Bernoulli distribution: $Z_{ij} \sim \mathrm{Bernoulli}(p)$.
2. $H$ is a $K \times K$ matrix with entries sampled from a multivariate Gaussian distribution.

I've defined a matrix product as: $ \kappa = ZHZ^T $

Given that $Z$ and $H$ are independent, I'm interested in computing $E\kappa^2$.

Any suggestions on how to approach this problem would be greatly appreciated. Thank you!

Update: I came across this paper that seems somewhat related to my problem, especially since I'm using an Indian buffet process prior $P(Z)=IBP(\alpha)$. However, the approximate posterior distribution applied for computation of expectation is a Bernoulli with variable $p$. Here's the progress I've made so far, but I'm uncertain of its correctness. I'd love to get feedback on whether this is a structured way to compute the expectation: I wrote $\mathrm{vec}(\kappa^2)=((Z \otimes Z) \mathrm{vec}(H) \otimes (Z \otimes Z) \mathrm{vec}(H)) \mathrm{vec}(I)$. Then I have $ \mathrm{vec}(H) \otimes \mathrm{vec}(H) = \mathrm{vec}(H \otimes H)$ \begin{equation} \mathrm{vec}(\kappa^2) = (Z \otimes Z \otimes Z \otimes Z) \mathrm{vec}(H \otimes H) \end{equation} The fourth order moment is \begin{equation} \begin{split} \mathbb{E}_{Z}\Big[Z \otimes Z \otimes Z \otimes Z\Big]&=\boldsymbol{p}\otimes\boldsymbol{p}\otimes\boldsymbol{p}\otimes\boldsymbol{p}+\mathfrak{S}_6\big[\mathrm{diag}(\boldsymbol{p}-\boldsymbol{p}^2)\otimes\boldsymbol{p}\otimes\boldsymbol{p}\big]\\ &+\mathfrak{S}_3\big[\mathrm{diag}(\boldsymbol{p}-\boldsymbol{p}^2)\otimes\mathrm{diag}(\boldsymbol{p}-\boldsymbol{p}^2)\big]\\ &+\mathfrak{S}_4\big[\mathrm{diag}\big(\boldsymbol{p}-3\boldsymbol{p}^2+2\boldsymbol{p}^3\big)\otimes\boldsymbol{p}\big]\\ &+\mathrm{diag}\big(\boldsymbol{p}-7\boldsymbol{p}^2+12\boldsymbol{p}^3-6\boldsymbol{p}^4\big) \end{split} \end{equation} where $\mathfrak{S}_3\big[A\big]_{ikj}=A_{ijk}+A_{jki}+A_{kij}$ and $\mathfrak{S}_6\big[A\big]_{ikj}=A_{ijk}+A_{kij}+A_{jki}+A_{jik}+A_{kji}+A_{ikj}$ if $A=B\otimes C$. I could not find the definition of $\mathfrak{S}_4$ in the paper. Could anyone clarify the definition of $\mathfrak{S}_4$ and how is this moment computed? Furthermore, is my fist computational trick on track? Thanks.

Answer 1

$\newcommand{\ka}{\kappa}\newcommand{\si}{\sigma}$We have $\ka^2=ZHZ^TZHZ^T$ and hence, for $i$ and $j$ in $[n]:=\{1,\dots,n\}$,
\begin{equation} (E\ka^2)_{ij}=\sum_{k,l,m,s,r}EZ_{ik}H_{kl}Z_{ml}Z_{ms}H_{sr}Z_{jr} \\ =\sum_{k,l,m,s,r}EZ_{ik}Z_{ml}Z_{ms}Z_{jr}\, EH_{kl}H_{sr}, \end{equation} since $Z$ and $H$ are independent. Assume $n\ge2$; the case $n=1$ is much easier.

As the $H_{kl}$'s are iid $N(\mu,\si^2)$ for some real $\mu$ and some real $\si>0$, we have
$EH_{kl}H_{sr}=\mu^2+\si^2\,1(s=k,r=l)$. So, \begin{equation} (E\ka^2)_{ij}=\mu^2 A_{ij}+\si^2 B_{ij}, \end{equation} where \begin{equation} A_{ij}:=\sum_{m\in[n]}EZ_{i\cdot}Z_{m\cdot}^2 Z_{j\cdot}, \quad Z_{i\cdot}:=\sum_{k\in[K]}Z_{ik}, \end{equation} \begin{equation} B_{ij}:=\sum_{k,l,m}EZ_{ik}Z_{ml}Z_{mk}Z_{jl}. \end{equation}

So, it remains to compute $A_{ij}$ and $B_{ij}$. Moreover, by symmetry, $A_{ij}$ and $B_{ij}$ depend only on whether $i=j$ or not.

So, it suffices to compute $A_{11}$, $A_{12}$, $B_{11}$, $B_{12}$.

Note that \begin{equation} A_{11}=(n-1)M_2^2+M_4,\quad A_{12}=(n-2)M_1^2 M_2+2M_3 M_1, \end{equation} where $M_r:=EZ_{1\cdot}^r$. Since $Z_{1\cdot}$ is a binomial random variable (r.v.) with parameters $k,p$, we easily find \begin{equation} A_{11}=K p \left((K-1) p^3 \left(K^2 n-K (n+4)+6\right) \\ +2 (K-1) p^2 (K (n+2)-6)+p (K (n+6)-7)+1\right), \end{equation} \begin{equation} A_{12}=K^2 p^2 \left((K-1) p^2 (K n-4)+p (K (n+4)-6)+2\right). \end{equation}

It remains to compute $B_{11}$, $B_{12}$. To do this, note first that \begin{equation} EZ_{ik}Z_{ml}Z_{mk}Z_{jl}=p^{\nu_{i,j,k,l,m}}, \end{equation} where $\nu_{i,j,k,l,m}$ is the cardinality (that is, the number of distinct elements) of the set \begin{equation} S_{i,j,k,l,m}:=\{(i,k),(m,l),(m,k),(j,l)\}. \end{equation} The number $\nu_{i,j,k,l,m}$ depends on $i,j,k,l,m$ in a somewhat complicated manner, so that we have to consider a number of cases.

For given $i,j,k,l,m$, let $f=f_{i,j,k,l,m}$ be the map from the set $[4]=\{1,2,3,4\}$ onto $S$ such that $f(1)=(i,k)$, $f(2)=(m,l)$, $f(3)=(m,k)$, $f(4)=(j,l)$. The function $f_{i,j,k,l,m}$ naturally generates the equivalence $\sim\;=\;\sim_{i,j,k,l,m}$ over the set $[4]$ given by the formula $a\sim b\iff f(a)=f(b)$ for $a,b$ in $[4]$. This equivalence partitions the set $[4]$ into equivalence classes. Then $\nu_{i,j,k,l,m}$ is the cardinality of this partition, that is, the number of equivalence classes for the equivalence $\sim_{i,j,k,l,m}$.

There are $15$ equivalences over the set $[4]$, and hence there are $15$ corresponding partitions of $[4]$. However, not all of these $15$ equivalences/partitions are generated by functions of the form $f_{i,j,k,l,m}$. Anyhow, for any equivalence, say $\approx$, of those $15$ equivalences, the condition $\approx\;=\;\sim_{i,j,k,l,m}$ is a (possibly empty) condition on $i,j,k,l,m$, which is expressible in terms of equalities between $i,j,k,l,m$. Thus, we get the contribution of each of the $15$ equivalences/partitions into the sum $B_{ij}$.

Thus we finally get \begin{equation} B_{11}=K p \left((K-1) (n-1) p^3+p (K+n-2)+1\right), \end{equation} \begin{equation} B_{12}=K p^2 ((K-1) p+1) ((n-2) p+2). \end{equation}

Details of the calculations are provided in this pdf image of a Mathematica notebook.

---

The more difficult part of these calculations is, of course, the calculations of $B_{11}$ and $B_{12}$, as they involve some combinatorics. The results of these calculations have been checked for some small values of $n$ and $K$.

Answer 2

If $H$ has iid $N(0,1)$ entries, write $\kappa=\langle Z^TZ, H\rangle$ using the usual Frobenius inner product $\langle A,B\rangle = trace[A^TB]$. Conditionally on $Z$, $\kappa\sim N(0,\|Z^TZ\|_F^2)$ where $\|\cdot\|_F$ is the Frobenius norm so that $E[\kappa^2]=E[\|Z^TZ\|_F^2]=\|E[Z^TZ]\|_F^2 + E[\|Z^TZ - E[Z^TZ]\|_F^2]$. The last two terms can be evaluated as a function of $\nu$ by treating the diagonal and off-diagonal entries separately.