Denote $a_n=a(n,t)$ and $b_n=b(n,t)$. To help avoiding the **min** function, write
$$b_n=\binom{n+3}3t^n+\sum_{j=0}^{n-1}\binom{3+j}3\left[t^{2n-j}+t^j\right].$$
Notice that $a_n=\frac{nt^{n+1}-(n+1)t^n+1}{(1-t)^2}$ and  $\sum_{j=0}^nt^j=\frac{1-t^{n+1}}{1-t}$. Your identity takes the form
$$(nt^{n+1}-(n+1)t^n+1)^2=(1-t)^3[(1-t^n)b_{n-1}-t(1-t^{n+1})b_{n-2}].$$
Now, as Mark Widon mentioned, try to read-off the coefficients of $t^k$.