Let us assume $\alpha\in[0,1)$. The case of $\alpha\in(-1,0]$ is similar. As $\sin B>0$ for $B\in (0,\pi)$, the inequality $f(A,B)<0$ amounts to $$ \alpha\sin A<\sin B-\sin(A+B),\quad -[\sin B+\sin(A+B)]<\alpha\sin A.\quad (\star) $$ Notice that $\sin A=2\sin\left(\frac{A}{2}\right)\cos\left(\frac{A}{2}\right)$, $\sin B-\sin(A+B)=-2\sin\left(\frac{A}{2}\right)\cos\left(\frac{A}{2}+B\right)$ and $\sin B+\sin(A+B)=2\cos\left(\frac{A}{2}\right)\sin\left(\frac{A}{2}+B\right)$. Substituting in $(\star)$ and cancelling the positive terms $2\sin\left(\frac{A}{2}\right)$ and $2\cos\left(\frac{A}{2}\right)$, we obtain the equivalent inequalities $$ \alpha\cos\left(\frac{A}{2}\right)<-\cos\left(\frac{A}{2}+B\right), \quad -\sin\left(\frac{A}{2}+B\right)<\alpha\sin\left(\frac{A}{2}\right).\quad (\star\star) $$ In $(\star\star)$, the LHS of the first inequality and the RHS of the second are non-negative. Hence $\frac{A}{2}+B$ - which belongs to $\left(0,\frac{3\pi}{2}\right)$ - must be in the second or the third quadrant; otherwise, the first inequality in $(\star\star)$ does not hold. Let us analyze these cases separately:
- If $\frac{\pi}{2}\leq\frac{A}{2}+B\leq\pi$, then the second inequality in $(\star\star)$ hold automatically (its RHS is non-negative); and the first one can be written as $$\alpha\cos\left(\frac{A}{2}\right)<\cos\left(\pi-\frac{A}{2}-B\right).$$ Applying the strictly decreasing function $\cos^{-1}:[0,1]\rightarrow\left[0,\frac{\pi}{2}\right]$ yields: $$\pi-\frac{A}{2}-B<\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right).$$ This of course implies $\frac{A}{2}+B\geq\frac{\pi}{2}$. But we also need $\frac{A}{2}+B\leq\pi$. Combining these, the bounds for $B$ in terms of $A\in(0,\pi)$ are given by $$ \pi-\frac{A}{2}-\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right)\leB<\pi-\frac{A}{2}. $$ The difference of the two bounds is $\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right)$. Consequently, the contribution to the area of $\{(A,B)\mid f(A,B)<0\}$ is $$ \int_{\{(A,B)\mid f(A,B)<0, \frac{\pi}{2}\leq\frac{A}{2}+B\leq\pi\}}\mathbf{1}= \int_{0}^\pi\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right){\rm{d}}A. $$
- If $\pi\leq\frac{A}{2}+B\leq\frac{3\pi}{2}$, all terms appearing in $(\star\star)$ are non-negative. We first rewrite these inequalities as $$ \alpha\cos\left(\frac{A}{2}\right)<\cos\left(\frac{A}{2}+B-\pi\right), \quad \sin\left(\frac{A}{2}+B-\pi\right)<\alpha\sin\left(\frac{A}{2}\right). $$ Next applying strictly monotonic functions $\cos^{-1}:[0,1]\rightarrow\left[0,\frac{\pi}{2}\right]$ and $\sin^{-1}:[0,1]\rightarrow\left[0,\frac{\pi}{2}\right]$ to them results in: $$ \frac{A}{2}+B-\pi<\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right),\quad \frac{A}{2}+B-\pi<\sin^{-1}\left(\alpha\sin\left(\frac{A}{2}\right)\right). $$ Hence the upper bound $$ B<\pi+\min\left\{\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right),\left(\alpha\sin\left(\frac{A}{2}\right)\right)\right\}-\frac{A}{2} $$ which of course implies $\frac{A}{2}+B\leq\frac{3\pi}{2}$. But $\pi\leq\frac{A}{2}+B$ is also required. We therefore arrive at the bounds for $B$ in terms of $A\in(0,\pi)$: $$ \pi-\frac{A}{2}\leq B\leq \pi+\min\left\{\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right),\left(\alpha\sin\left(\frac{A}{2}\right)\right)\right\}-\frac{A}{2}. $$