Denote $a_n=a(n,t)$ and $b_n=b(n,t)$. To help avoiding the **min** function, write
$$b_n=\binom{n+3}3t^n+\sum_{j=0}^{n-1}\binom{3+j}3\left[t^{2n-j}+t^j\right].$$
Notice that $a_n=\frac{nt^{n+1}-(n+1)t^n+1}{(1-t)^2}$ and  $\sum_{j=0}^nt^j=\frac{1-t^{n+1}}{1-t}$. Your identity takes the form
$$(nt^{n+1}-(n+1)t^n+1)^2=(1-t)^3[(1-t^n)b_{n-1}-t(1-t^{n+1})b_{n-2}].$$
Now, as Mark Widon mentioned, try to read-off the coefficients of $t^k$.

**UPDATE.** Resorting back the original formulation of the claim
$$a_n^2=b_{n-1}\sum_{j=0}^{n-1}t^j-t\,b_{n-2}\sum_{j=0}^nt^j,$$
I was able to rewrite (after lots of routine algebraic simplification and reorganization) this as
$$\left(\sum_{j=0}^{n-1}(j+1)t^j\right)^2
=\sum_{j=0}^{n-1}\binom{3+j}3t^j+\sum_{j=0}^{n-2}\left[\beta_n-\beta_{j+1}-\binom{n-j}3\right]\,t^{n+j}$$
where $\beta_k=\frac{k(k+1)(2k+1)}6$ (the sum of squares function).

Once we got this far, the next step is to compare the coefficients of $t^k$.