Well, if $X^3=1$, then $(X-1)(X^2+X+1)=0$. At first, assume that 1 is eigenvalue of $X$, then the second eigenvalue $v$ belongs to the field $GF(32)$, so $v^3=v^{31}=1$, $v=1$. That is, $X=Id+Y$ for nilpotent $Y$, $X^3=Id+3Y=Id+Y$, $Y=0$, order of $X$ equals 1, not 3. So $X^2+X+1=0$, and this is both minimal and characteristic polynomial of $X$. So, we count number of quadruples $(a,b,c,d)$ for which $a+d=1,ad-bc=1$. Fix arbitrary $d$ and arbitrary $c\ne 0$, then $a$ and $b$ are defined uniquely. Thus the answer is $32\cdot 31$.
In general case do the same steps, you have to know how many linear and quadratic factors does polynomial $X^t-1$ have in your field, this is standard (number of linear factors is $gcd(t,q-1)$, total degree of linear and quadratic factors is $gcd(t,q^2-1)$.)