This proof is similar to River Li's, but it requires no machine calculation. Also, we will show more, namely that the original polynomial has three real roots, the smallest of which is at most $$u:=\frac{5-\sqrt{15}}{10}=0.11270166\dotsc$$ This estimate is sharp, because in case of $\alpha=\beta=\gamma=\pi/2$, the polynomial is $$20t^3-30t^2+12t-1=20\left(t-\frac{1}{2}\right)\left(t-\frac{5-\sqrt{15}}{10}\right)\left(t-\frac{5+\sqrt{15}}{10}\right).$$ We turn to the proof. With the notation $$Q:=\frac{\cos^2\alpha+\cos^2\beta+\cos^2\gamma}{3} \qquad\text{and}\qquad R:=\cos\alpha\cos\beta\cos\gamma,$$ the polynomial in the original post equals $$p(t):=20t^3-30t^2+12(1-Q)t+(3Q-2R-1).$$ The coefficients of this cubic polynomial are $$a:=20,\qquad b:=-30,\qquad c:=12(1-Q),\qquad d:=3Q-2R-1,$$ hence its discriminant equals \begin{align*} \Delta\ := \ & 18 a b c d - 4 b^3 d + b^2 c^2 - 4 a c^3 - 27 a^2 d^2 \\ \ = \ & 2160 (1 + 12 Q + 3 Q^2 + 64 Q^3 - 60 Q R - 20 R^2). \end{align*} Here $2 QR\leq Q^2+R^2$, and also $R^2\leq Q^3$ by the AM-GM inequality, whence \begin{align*} \Delta/2160 \ & \geq \ 1 + 12 Q + 3 Q^2 + 64 Q^3 - 30 Q^2 - 50 R^2 \\ & \geq \ 1 + 12 Q - 27 Q^2 + 14 Q^3 \\ & = \ (Q-1)^2(1+14Q) \\ & \geq \ 0. \end{align*} So $p(t)$ has three real roots (counted with multiplicity), and the same is true of the shifted polynomial $$p(u+t)=20t^3-6\sqrt{15}t^2+6(1-2Q)t+\left(6\sqrt{\frac{3}{5}}Q-3Q-2R\right).$$ It remains to show that this new polynomial $p(u+t)$ has a nonpositive root $t$. If this is not the case, then $p(u+t)$ has three positive roots (counted with multiplicity), hence in particular $$1-2Q>0>6\sqrt{\frac{3}{5}}Q-3Q-2R.$$ However, this is a contradiction, because it would yield that $$Q^3\geq R^2>\left(3\sqrt{\frac{3}{5}}-\frac{3}{2}\right)^2 Q^2\geq\frac{1}{2}Q^2\geq Q^3.$$
