This may be a partial solution to your problem:

I claim that if there exists a shared eigenvector, $x$ of $A$ and $B$ with common eigenvalue of $1$ then $\det(AB - BA) = \det[A,B] = 0$.

Proof:

Suppose that there exists a shared eigenvector $x$ such that $Ax=x$ and $Bx=x$. Then, as Muro suggested, $ABx=x=BAx$. Hence, $(AB - BA)x = 0$ for some $x \neq 0$. This implies that the matrix $AB - BA = [A, B]$ is not invertible. So $\det([A,B]) = 0$.

I don't know if the condition is both necessary and sufficient. In particular, I do not know if converse to the statement above is true.

This may be a partial solution to your problem:

I claim that if there exists a shared eigenvector, $x$ of $A$ and $B$ with common eigenvalue of $1$ then $\det(AB - BA) = \det[A,B] = 0$.

Proof:

Suppose that there exists a shared eigenvector $x$ such that $Ax=x$ and $Bx=x$. Then, as Muro suggested, $ABx=x=BAx$. Hence, $(AB - BA)x = 0$ for some $x \neq 0$. This implies that the matrix $AB - BA = [A, B]$ is not invertible. So $\det([A,B]) = 0$.

I don't know if the condition is both necessary and sufficient. In particular, I do not know if converse to the statement above is true.

(Edit) I forgot to mention that if $A$ and $B$ commute, then they share a common eigenvector. This is a standard exercise in linear algebra.