The identity is not correct as written (as you can check by evaluating both sides for some small $a$ and $b$). It becomes correct if the factor $(-1)^k$ is omitted. To see this, note that it will be enough to check the identity when $q$ is a prime power, in which case we can choose a field $F$ with $|F|=q$, and vector spaces $A$ and $B$ of dimension $a$ and $b$ over $F$. The left hand side is $|\text{Hom}(A,B)|$. To give a homomorphism $\alpha\colon A\to B$ we have to choose a rank $k$, a kernel $U\leq A$ of dimension $a-k$, an image $V\leq B$ of dimension $k$, and an isomorphism $\alpha_1\colon A/U\to V$. The number of choices for $U$ is $\binom{a}{a-k}_q=\binom{a}{k}_q$. The number of choices for $V$ is $\binom{b}{k}_q$. The number of choices for $\alpha_1$ is $|GL_k(F)|=q^{\binom{k}{2}}(q;q)_k$.