The answer is yes, and this follows from the comments, but it's worth spelling out the (non-obvious) details.
By Will Sawin's comment, the claim will follow if we can show that $$ \dim_\mathbb{Q} H^p(B;H^q(F;\mathbb{Q})) = \dim_\mathbb{Q} H_p(B;H_q(F;\mathbb{Q})) $$ for all $p,q\ge0$. The Universal Coefficient Theorem does not hold with twisted coefficients. However, note that the vector spaces on the right are by definition the homology groups of the chain complex $$ C_*(\widetilde{B})\otimes_\pi H_q(X;\mathbb{Q}), $$ with differential induced by that of the chain complex $C_*(\widetilde{B})$ of the universal cover. Here $\pi=\pi_1(B)$ and we are taking tensor product of $\mathbb{Z}\pi$-modules.
Following archipelago's comment, let's see what happens if we dualize this chain complex by taking Hom into the rationals. By the tensor-hom adjunction there are natural isomorphisms $$ {\rm Hom}_\mathbb{Z}\big(C_*(\widetilde{B})\otimes_\pi H_q(X;\mathbb{Q}),\mathbb{Q}\big)\cong {\rm Hom}_\pi\big(C_*(\widetilde{B}),{\rm Hom}_\mathbb{Z}(H_q(X;\mathbb{Q}),\mathbb{Q})\big) \cong {\rm Hom}_\pi\big(C_*(\widetilde{B}), H^q(F;\mathbb{Q})\big),$$ and so we get precisely the cochain complex whose cohomology gives the vector spaces on the left. Since everything in sight is a rational vector space, the usual algebraic Universal Coefficient Theorem implies the claim.