I confirm Peter's suggestion that the best constant is $\frac1{\sqrt2}$. For let $\omega$ be this best constant. Then the inequality amounts to writing
$$3-{\rm Tr}(A^tB^tAB)\le2\omega^2(3-{\rm Tr}\,A)(3-{\rm Tr}\,B).$$
Let me parametrize $A$ by the angle of rotation $\theta$ and the axis of rotation $u$, a unit vector. Likewise $B$ has an angle of rotation $\alpha$ and an axis $v$. The left-hand side above is $8\omega^2(1-c)(1-c')$, where $c=\cos\theta$ and $c'=\cos\alpha$.

Now $B^tAB$ is the rotation of same angle $\theta$ about $w=B^{-1}u$, and $A^t$ the rotation of angle $-\theta$ about $u$. It is an instructive calculus that
$${\rm Tr}(A^tB^tAB)=2c+c^2+(1-c)^2(w\cdot u)^2+2s^2w\cdot u.$$
Because of $w\cdot u\ge c'$, we find that $\omega$ is the best constant in
$$3-2c-c^2+(1-c)^2c'^2-2s^2c'\le8\omega^2(1-c)(1-c').$$
The left-hand side factorizes as $(1-c)(1-c')(3+c+c'-cc')$, and there remains the inequality
$$3+c+c'-cc'\le8\omega^2.$$
Therefore $\omega^2$ is the supremum of $\frac18(3+c+c'-cc')$ over $[-1,1]^2$. Because this quantity if affine in both $c$ and $c'$, the supremum is achieved at a vertex, and its value is $4$. Whence $\omega^2=\frac48=\frac12$.