[Edit: Now I answer all questions.]

The answer to the first question is yes, the answer to the second question is no, and the answer to the third question is if and only if $p \geq 2$ (only a guess in the case $p<2$).

• First question

The inequality $$ \mathbb E \|A G B\|_* \leq \min( \|A\|_* \|B\|_F,\|A\|_F \|B\|_*)$$ always holds. By symmetry, it is enough to prove the inequality $\mathbb E \|A G B\|_* \leq \|A\|_* \|B\|_F$. And by writing $A$ as a sum of rank one operators ("the unit ball of the trace class is the convex hull of the norm $1$ rank $1$ matrices"), we can assume that $A$ has rank one.

In that case, $A G B$ has rank one for every $G$. Using (1) that for a rank $1$ matrix, the trace norm and Frobenius norm coincide, and (2) that the $L^1$ norm is less than the $L^2$ norm, we get $$ \mathbb E \|A G B\|_* = \mathbb E \|A G B\|_F \leq (\mathbb E \|A G B\|_F^{2})^{\frac 1 2} = \|A\|_F \|B\|_F=\|A\|_* \|B\|_F.$$ The second equality is a straighforward computation, at least when $A$ and $B$ are diagonal.

• Second question, and third question when $p \geq 2$.

I follows from the non-commutative Khintchine inequalities of Françoise Lust-Picard that, for every $2 \leq p < \infty$, $$ \mathbb E \|A G B\|_p \simeq \max( \|A\|_p \|B\|_2,\|A\|_2 \|B\|_p)\ if\ p \geq 2$$ (up to constants depending on $p$, whose growth rate is known as $p \to \infty$).

Indeed, the non-commutative Khintchine inequalities states that if $(C_k)$ is a family of matrices and $g_k$ are iid $N(0,1)$ random variables, $$ \mathbb E \| \sum_k g_k C_k\|_p \simeq \max ( \|(\sum C_k C_k^*)^{\frac 1 2}\|_p, \|(\sum C_k^* C_k)^{\frac 1 2}\|_p).$$ (the inequality is sometimes stated in terms of Bernoulli random variables instead of gaussians, and with $L^p$-norm, of $L^2$ norm instead of $L^1$-norm on the left-hand side, but all versions are equivalent by Kahane's inequalities and standard probabilistic arguments). Applying this to $C_{i,j}$, the product of the $i$-th column of $A$ and and $j$-th row of $B$, one gets the answer because $\sum_{i,j} C_{i,j} C_{i,j}^* = \|B\|_2^2 A A^*$ and $\sum_{i,j} C_{i,j}^* C_{i,j}=\|A\|_2^2 B^*B$.

• Third question if $p \leq 2$.

Here also, the non-commutative Khintchine inequalities (due to Lust-Picard and Pisier when $p=1$) allow, in principle, to give an explicit equivalent of $\mathbb E\|A G B\|_p$. Indeed, when $p \leq 2$, the inequality reads $$ \mathbb E \| \sum_k g_k C_k\|_p \simeq \inf \|(\sum D_k D_k^*)^{\frac 1 2}\|_p+ \|(\sum E_k^* E_k)^{\frac 1 2}\|_p),$$ (up to universal constants) where the infimum is over all families $(D_k,E_k)$ such that $C_k=D_k+E_k$ for all $k$.

This gives a very involved proof of the inequality of the first question. I believe that it can be deduced from this inequality that the inequality $$ \mathbb E \|A G B\|_* \geq \min( \|A\|_* \|B\|_F,\|A\|_F \|B\|_*)$$ does not hold if $p<2$, but this deserves to be checked.

[Edit: Now I answer all questions.]

The answer to the first question is yes, the answer to the second question is no, and the answer to the third question is if and only if $p \geq 2$ (only a guess in the case $p<2$).

• First question

The inequality $$ \mathbb E \|A G B\|_* \leq \min( \|A\|_* \|B\|_F,\|A\|_F \|B\|_*)$$ always holds. By symmetry, it is enough to prove the inequality $\mathbb E \|A G B\|_* \leq \|A\|_* \|B\|_F$. And by writing $A$ as a sum of rank one operators ("the unit ball of the trace class is the convex hull of the norm $1$ rank $1$ matrices"), we can assume that $A$ has rank one.

In that case, $A G B$ has rank one for every $G$. Using (1) that for a rank $1$ matrix, the trace norm and Frobenius norm coincide, and (2) that the $L^1$ norm is less than the $L^2$ norm, we get $$ \mathbb E \|A G B\|_* = \mathbb E \|A G B\|_F \leq (\mathbb E \|A G B\|_F^{2})^{\frac 1 2} = \|A\|_F \|B\|_F=\|A\|_* \|B\|_F.$$ The second equality is a straighforward computation, at least when $A$ and $B$ are diagonal.

• Second question, and third question when $p \geq 2$.

It follows from the non-commutative Khintchine inequalities of Françoise Lust-Picard that, for every $2 \leq p < \infty$, $$ \mathbb E \|A G B\|_p \simeq \max( \|A\|_p \|B\|_2,\|A\|_2 \|B\|_p)\ if\ p \geq 2$$ (up to constants depending on $p$, whose growth rate is known as $p \to \infty$).

Indeed, the non-commutative Khintchine inequalities states that if $(C_k)$ is a family of matrices and $g_k$ are iid $N(0,1)$ random variables, $$ \mathbb E \| \sum_k g_k C_k\|_p \simeq \max ( \|(\sum C_k C_k^*)^{\frac 1 2}\|_p, \|(\sum C_k^* C_k)^{\frac 1 2}\|_p).$$ (the inequality is sometimes stated in terms of Bernoulli random variables instead of gaussians, and with $L^p$-norm, of $L^2$ norm instead of $L^1$-norm on the left-hand side, but all versions are equivalent by Kahane's inequalities and standard probabilistic arguments). Applying this to $C_{i,j}$, the product of the $i$-th column of $A$ and and $j$-th row of $B$, one gets the answer because $\sum_{i,j} C_{i,j} C_{i,j}^* = \|B\|_2^2 A A^*$ and $\sum_{i,j} C_{i,j}^* C_{i,j}=\|A\|_2^2 B^*B$.

• Third question if $p \leq 2$.

Here also, the non-commutative Khintchine inequalities (due to Lust-Picard and Pisier when $p=1$) allow, in principle, to give an explicit equivalent of $\mathbb E\|A G B\|_p$. Indeed, when $p \leq 2$, the inequality reads $$ \mathbb E \| \sum_k g_k C_k\|_p \simeq \inf \|(\sum D_k D_k^*)^{\frac 1 2}\|_p+ \|(\sum E_k^* E_k)^{\frac 1 2}\|_p),$$ (up to universal constants) where the infimum is over all families $(D_k,E_k)$ such that $C_k=D_k+E_k$ for all $k$.

This gives a very involved proof of the inequality of the first question. I believe that it can be deduced from this inequality that the inequality $$ \mathbb E \|A G B\|_p \geq \min( \|A\|_p \|B\|_2,\|A\|_2 \|B\|_p)$$ does not hold if $p<2$, but this deserves to be checked.

The inequality $$ \mathbb E \|A G B\|_* \leq \min( \|A\|_* \|B\|_F,\|A\|_F \|B\|_*)$$ always holds. By symmetry, it is enough to prove the inequality $\mathbb E \|A G B\|_* \leq \|A\|_* \|B\|_F$. And by writing $A$ as a sum of rank one operators ("the unit ball of the trace class is the convex hull of the norm $1$ rank $1$ matrices"), we can assume that $A$ has rank one.

In that case, $A G B$ has rank one for every $G$. Using (1) that for a rank $1$ matrix, the trace norm and Frobenius norm coincide, and (2) that the $L^1$ norm is less than the $L^2$ norm, we get $$ \mathbb E \|A G B\|_* = \mathbb E \|A G B\|_F \leq (\mathbb E \|A G B\|_F^{2})^{\frac 1 2} = \|A\|_F \|B\|_F=\|A\|_* \|B\|_F.$$ The second equality is a straighforward computation, at least when $A$ and $B$ are diagonal.

[Edit: Now I answer all questions.]

The answer to the first question is yes, the answer to the second question is no, and the answer to the third question is if and only if $p \geq 2$ (only a guess in the case $p<2$).

• First question

The inequality $$ \mathbb E \|A G B\|_* \leq \min( \|A\|_* \|B\|_F,\|A\|_F \|B\|_*)$$ always holds. By symmetry, it is enough to prove the inequality $\mathbb E \|A G B\|_* \leq \|A\|_* \|B\|_F$. And by writing $A$ as a sum of rank one operators ("the unit ball of the trace class is the convex hull of the norm $1$ rank $1$ matrices"), we can assume that $A$ has rank one.

In that case, $A G B$ has rank one for every $G$. Using (1) that for a rank $1$ matrix, the trace norm and Frobenius norm coincide, and (2) that the $L^1$ norm is less than the $L^2$ norm, we get $$ \mathbb E \|A G B\|_* = \mathbb E \|A G B\|_F \leq (\mathbb E \|A G B\|_F^{2})^{\frac 1 2} = \|A\|_F \|B\|_F=\|A\|_* \|B\|_F.$$ The second equality is a straighforward computation, at least when $A$ and $B$ are diagonal.

• Second question, and third question when $p \geq 2$.

I follows from the non-commutative Khintchine inequalities of Françoise Lust-Picard that, for every $2 \leq p < \infty$, $$ \mathbb E \|A G B\|_p \simeq \max( \|A\|_p \|B\|_2,\|A\|_2 \|B\|_p)\ if\ p \geq 2$$ (up to constants depending on $p$, whose growth rate is known as $p \to \infty$).

Indeed, the non-commutative Khintchine inequalities states that if $(C_k)$ is a family of matrices and $g_k$ are iid $N(0,1)$ random variables, $$ \mathbb E \| \sum_k g_k C_k\|_p \simeq \max ( \|(\sum C_k C_k^*)^{\frac 1 2}\|_p, \|(\sum C_k^* C_k)^{\frac 1 2}\|_p).$$ (the inequality is sometimes stated in terms of Bernoulli random variables instead of gaussians, and with $L^p$-norm, of $L^2$ norm instead of $L^1$-norm on the left-hand side, but all versions are equivalent by Kahane's inequalities and standard probabilistic arguments). Applying this to $C_{i,j}$, the product of the $i$-th column of $A$ and and $j$-th row of $B$, one gets the answer because $\sum_{i,j} C_{i,j} C_{i,j}^* = \|B\|_2^2 A A^*$ and $\sum_{i,j} C_{i,j}^* C_{i,j}=\|A\|_2^2 B^*B$.

• Third question if $p \leq 2$.

Here also, the non-commutative Khintchine inequalities (due to Lust-Picard and Pisier when $p=1$) allow, in principle, to give an explicit equivalent of $\mathbb E\|A G B\|_p$. Indeed, when $p \leq 2$, the inequality reads $$ \mathbb E \| \sum_k g_k C_k\|_p \simeq \inf \|(\sum D_k D_k^*)^{\frac 1 2}\|_p+ \|(\sum E_k^* E_k)^{\frac 1 2}\|_p),$$ (up to universal constants) where the infimum is over all families $(D_k,E_k)$ such that $C_k=D_k+E_k$ for all $k$.

This gives a very involved proof of the inequality of the first question. I believe that it can be deduced from this inequality that the inequality $$ \mathbb E \|A G B\|_* \geq \min( \|A\|_* \|B\|_F,\|A\|_F \|B\|_*)$$ does not hold if $p<2$, but this deserves to be checked.