I can prove a related result. Perhaps someone can
modify the proof to solve Dominic's problem.

I use multivariate notation such as $x^\alpha=x_1^{\alpha_1}\cdots
x_n^{\alpha_m}$, where $\alpha=(\alpha_1,\dots,\alpha_m)$. Let
$\alpha\in \{0,1,2,\dots\}^m$ and $\beta\in\{0,1,2,\dots\}^n$. Let $f(\alpha,\beta)$ be the
number of $m\times n$ matrices with entries $0,1,2$,
and with each entry equal to 1 colored either red or blue, with row
sum vector $\alpha$ and column sum vector $\beta$.

**Theorem.**
   $$ f(\alpha,\beta) \leq f((n,n,\dots,n),(m,m,\dots,m)). $$

*Proof.* Let $g(\alpha,\beta)$ be the
number of $m\times n$ matrices with entries $-2,0,2$,
and with each entry equal to 0 colored either red or blue, with row
sum vector $\alpha$ and column sum vector $\beta$. By dividing each
entry of such a matrix by 2 and then adding 1, it is clear that
 $$ f(\alpha,\beta)=g\left(2\alpha-2(n,\dots,n),
    2\beta-2(m,\dots,m)\right). $$ 
Hence we want to show that
  $$ g(\alpha,\beta) \leq g((0,0,\dots,0),(0,0,\dots,0)). $$

We have for fixed $m,n$ that
  $$ \sum_{\alpha,\beta} g(\alpha,\beta)x^\alpha y^\beta =
      \prod_{r=1}^m\prod_{s=1}^n (x_r^{-1}y_s^{-1}+x_ry_s)^2. $$
Since for any integer $k$ we have $\int_0^{2\pi}e^{ikx}dx = 1$ if $k=
  0$ and otherwise is $0$, it follows that
  $$ g(\alpha,\beta) = \frac{1}{(2\pi)^{m+n}}
    \int_0^{2\pi}\cdots \int_0^{2\pi} e^{-i(\alpha_1 \theta_1+\cdots+
     \alpha_m\theta_m+\beta_1\psi_1+\cdots+\beta_n\psi_n)}\\
     \prod_{r=1}^m\prod_{s=1}^n (e^{-i(\theta_r+\psi_s)}
     +e^{i(\theta_r+\psi_s)})^2\,d\theta\,d\psi. $$
Now $(e^{-i(\theta_r+\psi_s)}+e^{i(\theta_r+\psi_s)})^2$ is a
nonnegative real number. Hence by the triangle inequality,
  \begin{eqnarray*} g(\alpha,\beta) & \leq &
      \frac{1}{(2\pi)^{m+n}}
    \int_0^{2\pi}\cdots \int_0^{2\pi}
     \prod_{r,s=1}^n (e^{-i(\theta_r+\psi_s)}
     +e^{i(\theta_r+\psi_s)})^2\,d\theta\,d\psi\\ & = &
     g((0,\dots,0),(0,\dots,0)).\ \Box \end{eqnarray*}