Let us assume $\alpha\in[0,1)$ (the case of $\alpha\in(-1,0]$ is similar). As $\sin B>0$ for $B\in (0,\pi)$, the inequality $f(A,B)<0$ amounts to
$$
\alpha\sin A<\sin B-\sin(A+B),\quad -[\sin B+\sin(A+B)]<\alpha\sin A.\quad (\star)
$$
Notice that
$\sin A=2\sin\left(\frac{A}{2}\right)\cos\left(\frac{A}{2}\right)$,
$\sin B-\sin(A+B)=-2\sin\left(\frac{A}{2}\right)\cos\left(\frac{A}{2}+B\right)$ and
$\sin B+\sin(A+B)=2\cos\left(\frac{A}{2}\right)\sin\left(\frac{A}{2}+B\right)$.
Substituting in $(\star)$ and cancelling the positive terms
$2\sin\left(\frac{A}{2}\right)$ and $2\cos\left(\frac{A}{2}\right)$, we obtain the equivalent inequalities
$$
\alpha\cos\left(\frac{A}{2}\right)<-\cos\left(\frac{A}{2}+B\right), \quad
-\sin\left(\frac{A}{2}+B\right)<\alpha\sin\left(\frac{A}{2}\right).\quad (\star\star)
$$
In $(\star\star)$, the LHS of the first inequality and the RHS of the second are non-negative. Hence $\frac{A}{2}+B$ - which belongs to $\left(0,\frac{3\pi}{2}\right)$ - must be in the second or the third quadrant; otherwise, the first inequality in $(\star\star)$ does not hold. Let us analyze these cases separately:
- If $\frac{\pi}{2}\leq\frac{A}{2}+B\leq\pi$, then the second inequality in $(\star\star)$ holds automatically (its RHS is always non-negative); and the first one can be written as
$$\alpha\cos\left(\frac{A}{2}\right)<\cos\left(\pi-\frac{A}{2}-B\right).$$ Applying the strictly decreasing function $\cos^{-1}:[0,1]\rightarrow\left[0,\frac{\pi}{2}\right]$ yields:
$$\pi-\frac{A}{2}-B<\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right).$$
This of course implies $\frac{A}{2}+B\geq\frac{\pi}{2}$. But we also need $\frac{A}{2}+B\leq\pi$. Combining these, the bounds for $B$ in terms of $A\in(0,\pi)$ are given by
$$
\pi-\frac{A}{2}-\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right)\leq B\leq\pi-\frac{A}{2}.
$$
The difference of the two bounds is $\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right)$. Consequently, the contribution to the area of
$\{(A,B)\mid f(A,B)<0\}$ is
$$
\int_{\{(A,B)\mid f(A,B)<0,\, \frac{\pi}{2}\leq\frac{A}{2}+B\leq\pi\}}\mathbf{1}=
\int_{0}^\pi\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right){\rm{d}}A.\quad (1)
$$
- If $\pi\leq\frac{A}{2}+B\leq\frac{3\pi}{2}$, all terms appearing in $(\star\star)$ are non-negative. We first rewrite these inequalities as
$$
\alpha\cos\left(\frac{A}{2}\right)<\cos\left(\frac{A}{2}+B-\pi\right), \quad
\sin\left(\frac{A}{2}+B-\pi\right)<\alpha\sin\left(\frac{A}{2}\right).
$$
Next applying strictly monotonic functions
$\cos^{-1}:[0,1]\rightarrow\left[0,\frac{\pi}{2}\right]$
and $\sin^{-1}:[0,1]\rightarrow\left[0,\frac{\pi}{2}\right]$ to them results in:
$$
\frac{A}{2}+B-\pi<\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right),\quad
\frac{A}{2}+B-\pi<\sin^{-1}\left(\alpha\sin\left(\frac{A}{2}\right)\right).
$$
Hence the upper bound
$$
B<\pi+\min\left\{\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right),\sin^{-1}\left(\alpha\sin\left(\frac{A}{2}\right)\right)\right\}-\frac{A}{2}
$$
which of course implies $\frac{A}{2}+B\leq\frac{3\pi}{2}$. But $\pi\leq\frac{A}{2}+B$ is also required. We therefore arrive at the bounds for $B$ in terms of $A\in(0,\pi)$:
$$
\pi-\frac{A}{2}\leq B\leq
\pi+\min\left\{\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right),\sin^{-1}\left(\alpha\sin\left(\frac{A}{2}\right)\right)\right\}-\frac{A}{2}.
$$
The difference of the bounds is $\min\left\{\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right),\sin^{-1}\left(\alpha\sin\left(\frac{A}{2}\right)\right)\right\}$. Therefore, the contribution to the area of
$\{(A,B)\mid f(A,B)<0\}$ is
$$
\int_{\{(A,B)\mid f(A,B)<0,\, \pi\leq\frac{A}{2}+B\leq\frac{3\pi}{2}\}}\mathbf{1}\\
=\int_{0}^\pi\min\left\{\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right),\sin^{-1}\left(\alpha\sin\left(\frac{A}{2}\right)\right)\right\}{\rm{d}}A.\quad (2)
$$
Adding $(1)$ and $(2)$, the area of $\{(A,B)\mid f(A,B)<0\}$ turns out to be
$$
\int_{0}^\pi\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right){\rm{d}}A\\+
\int_{0}^\pi\min\left\{\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right),\sin^{-1}\left(\alpha\sin\left(\frac{A}{2}\right)\right)\right\}{\rm{d}}A.\quad
(\star\star\star)$$
So the question is if the quantity above coincides with $\frac{\pi^2}{2}$ for all $\alpha\in [0,1)$. First, we claim that the minimum above is
$\sin^{-1}\left(\alpha\sin\left(\frac{A}{2}\right)\right)$. Notice that:
$$
\sin^{-1}\left(\alpha\sin\left(\frac{A}{2}\right)\right)
\leq\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right)\\
\Leftrightarrow
\cos^{-1}\left(\alpha\sin\left(\frac{A}{2}\right)\right)+\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right)\geq\frac{\pi}{2};
$$
and the cosine of the last angle appearing above is
$$
\left[\alpha\sin\left(\frac{A}{2}\right)\right]\left[\alpha\cos\left(\frac{A}{2}\right)\right]
-\sqrt{1-\alpha^2\sin^2\left(\frac{A}{2}\right)}
\sqrt{1-\alpha^2\cos^2\left(\frac{A}{2}\right)};$$
which is negative as
$\alpha\sin\left(\frac{A}{2}\right)<\sqrt{1-\alpha^2\cos^2\left(\frac{A}{2}\right)}$ and
$\alpha\cos\left(\frac{A}{2}\right)<\sqrt{1-\alpha^2\sin^2\left(\frac{A}{2}\right)}$ due to $|\alpha|<1$. We conclude that $(\star\star\star)$ is equal to
$$
\int_{0}^\pi\cos^{-1}\left(\alpha\cos\left(\frac{A}{2}\right)\right){\rm{d}}A+
\int_{0}^\pi\sin^{-1}\left(\alpha\sin\left(\frac{A}{2}\right)\right){\rm{d}}A.
$$
Call the expression above $h(\alpha)$. The goal is to establish $h(\alpha)=\frac{\pi^2}{2}$ for any $\alpha\in[0,1]$. This is clear when $\alpha=0$, and so it suffices to show $\frac{{\rm{d}}h}{{\rm{d}}\alpha}\equiv 0$. One has
$$
\frac{{\rm{d}}h}{{\rm{d}}\alpha}=
-\int_0^{\pi}\frac{\cos\left(\frac{A}{2}\right)}{\sqrt{1-\alpha^2\cos^2\left(\frac{A}{2}\right)}}{\rm{d}}A
+\int_0^{\pi}\frac{\sin\left(\frac{A}{2}\right)}{\sqrt{1-\alpha^2\sin^2\left(\frac{A}{2}\right)}}{\rm{d}}A;
$$
which is clearly zero because the change of variable $A\mapsto\pi-A$ indicates
$$\int_0^{\pi}\frac{\cos\left(\frac{A}{2}\right)}{\sqrt{1-\alpha^2\cos^2\left(\frac{A}{2}\right)}}{\rm{d}}A
=\int_0^{\pi}\frac{\sin\left(\frac{A}{2}\right)}{\sqrt{1-\alpha^2\sin^2\left(\frac{A}{2}\right)}}{\rm{d}}A.$$
This concludes the proof.
\lvert \rvert
for absolute value, not\mid \mid
. Compare $\lvert\alpha\rvert < 1$\lvert\alpha\rvert < 1
to $\mid\alpha\mid < 1$\mid\alpha\mid < 1
. I have edited accordingly. $\endgroup$