How to prove this inequality $$\left(a+\frac{1}{2} \left(a b-\sqrt{a^2-1} \sqrt{b^2-1}\right)\right)^{3/4}-\frac{\sqrt{3} \cos\left[\frac{3 (\pi -t)}{4}\right]}{2 \left(\frac{1}{2}+b\right)^{1/4}}-(a+\cos t)^{3/4}\ge 0$$ for $a,b\ge 1$ and $t\in[0,\pi]$. It can be noticed that for $a=b$ and $t=\pi/3$ the quantity is equal to zero.
This inequality would imply some sharp form of M. Riesz conjugate function theorem.
One can proceed by trying to simplify the inequality and eliminate variables by making various substitutions and estimates, so long as one doesn't concede too much of an error in the estimates. My first few attempts conceded too much and led to inequalities that were simply false, but eventually I was able to find a (rather complicated) argument that managed to reach the finish line, given below. There may be a simpler way to proceed, but I got the sense that there wasn't much "slack" in this inequality and so one does have to proceed with a certain amount of care.
The presence of the $\sqrt{a^2-1}$, $\sqrt{b^2-1}$ terms suggests hyperbolic substitution. Making the change of variables $$ a = \cosh \alpha; \quad b = \cosh(\alpha - \gamma)$$ with $0, \gamma \leq \alpha$, we have $$ ab - \sqrt{a^2-1} \sqrt{b^2-1} = \cosh \alpha \cosh(\alpha-\gamma) - \sinh \alpha \sinh(\alpha-\gamma) = \cosh \gamma$$ and so the inequality now becomes \begin{align*} &(\cosh \alpha + \frac{1}{2} \cosh \gamma)^{3/4} - (\cosh \alpha + \cos t)^{3/4} \\ &\quad \geq \frac{\sqrt{3}}{2} (\cosh(\alpha-\gamma)+\frac{1}{2})^{-1/4} \cos \frac{3(\pi-t)}{4}. \end{align*} The equality case is now $\gamma=0, t = \pi/3$. Note the faint resemblance to some sort of mean value theorem applied to the function $u \mapsto u^{3/4}$.
Now we replace the trigonometric expressions in $t$ with quadratic expressions in $x = \frac{1}{2}-\cos t$, which is a convenient further change of variable (in particular, it allows us to shift the equality case from $t=\pi/3$ to $x=0$, and eliminate $x$ shortly afterwards). One can verify numerically that
$$ \frac{\sqrt{3}}{2} \cos \frac{3(\pi-t)}{4} \leq \frac{3}{4} ( (\frac{1}{2}-\cos t) - \sigma (\frac{1}{2}-\cos t)^2 )$$
for $0 \leq t \leq \pi$, where the optimized constant
$$ \sigma := \frac{2}{3} - \frac{8}{9\sqrt{3}} \approx 0.153466\dots$$
is chosen so that we have equality at $t=\pi$ as well as $t=\pi/3$:
As it turns out, the precise numerical value of $\sigma$ will actually be surprisingly important in our arguments.
Making the change of variables $x = \frac{1}{2}-\cos t$ (so that $-\frac{1}{2} \leq x \leq \frac{3}{2}$), it now suffices to show that \begin{align*} & (\cosh \alpha + \frac{1}{2} \cosh \gamma)^{3/4} - (\cosh \alpha + \frac{1}{2}-x)^{3/4} \\ &\geq \frac{3}{4} (\cosh(\alpha-\gamma)+\frac{1}{2})^{-1/4} (x - \sigma x^2). \end{align*} The equality case is now $\gamma=x=0$, and the inequality begins to resemble the mean-value theorem even more.
From concavity of $u \mapsto u^{3/4}$ we have $$ (\cosh \alpha + \frac{1}{2}-x)^{3/4} \leq (\cosh \alpha + \frac{1}{2})^{3/4} - \frac{3}{4} (\cosh \alpha + \frac{1}{2})^{-1/4} x$$ and from the mean value theorem we have $$ (\cosh \alpha + \frac{1}{2} \cosh \gamma)^{3/4} - (\cosh \alpha + \frac{1}{2})^{3/4} \geq \frac{3}{4} (\cosh \alpha + \frac{1}{2} \cosh \gamma)^{-1/4} \frac{\cosh \gamma - 1}{2}$$ so after a little rearranging (and eliminating the common factor of $3/4$), it will suffice to show that \begin{align*} &\sigma (\cosh(\alpha-\gamma)+\frac{1}{2})^{-1/4} x^2 - ((\cosh(\alpha-\gamma)+\frac{1}{2})^{-1/4} - (\cosh \alpha + \frac{1}{2})^{-1/4}) x \\ & \quad + (\cosh \alpha + \frac{1}{2} \cosh \gamma)^{-1/4} \frac{\cosh \gamma - 1}{2} \geq 0. \end{align*} The left-hand side is quadratic in $x$ with positive leading coefficient, so it suffices to show that the discriminant is non-positive: \begin{align*} &2 \sigma (\cosh(\alpha-\gamma)+\frac{1}{2})^{-1/4} (\cosh \alpha + \frac{1}{2} \cosh \gamma)^{-1/4} (\cosh \gamma - 1) \\ &\quad \geq ((\cosh(\alpha-\gamma)+\frac{1}{2})^{-1/4} - (\cosh \alpha + \frac{1}{2})^{-1/4})^2. \end{align*} The equality case is $\gamma=0$.
Having eliminated $x$, the next task is to eliminate the role of $\alpha$, which we shall do by factoring out the quantity $$ A := \cosh(\alpha)+\frac{1}{2}.$$ It turns out that we need to split into the $\gamma \leq 0$ and $0 \leq \gamma \leq \alpha$ cases. If $\gamma \leq 0$, then $$ A \leq \cosh(\alpha-\gamma)+\frac{1}{2} \leq e^{|\gamma|} A$$ and thus $$ |(\cosh(\alpha-\gamma)+\frac{1}{2})^{-1/4} - (\cosh \alpha + \frac{1}{2})^{-1/4}| \leq (1-e^{-|\gamma|/4}) A^{-1/4};$$ also $$ \cosh(\alpha-\gamma)+\frac{1}{2}, \cosh(\alpha)+\frac{1}{2} \cosh \gamma \leq e^{|\gamma|} A$$ and so (after cancelling out the factors of $A$) it will suffice to show that $$ 2 \sigma e^{-|\gamma|/2} (\cosh \gamma - 1) \geq (1-e^{-|\gamma|/4})^2$$ or equivalently $$ 2 \sigma (\cosh \gamma - 1) \geq (e^{|\gamma|/4}-1)^2. \tag{ (1)}$$ If instead $0 \leq \gamma \leq \alpha$, we observe that $$ \cosh(\alpha-\gamma) \cosh(\gamma) = \cosh(\alpha) - \sinh(\alpha-\gamma) \sinh(\gamma) \leq \cosh(\alpha)$$ and thus \begin{align*} (\cosh(\alpha-\gamma)&+\frac{1}{2}) (\cosh(\alpha)+\frac{1}{2} \cosh \gamma)\\ &= \cosh(\alpha-\gamma) \cosh(\alpha) + \frac{\cosh(\alpha)+\cosh(\alpha-\gamma)\cosh(\gamma)}{2} + \frac{\cosh(\gamma)}{4} \\ &\leq e^{\gamma} \cosh^2(\alpha) + \cosh(\alpha) + \frac{e^\gamma}{4} \\ &\leq e^\gamma A^2 \end{align*} and also $$ e^{-\gamma} A \leq \cosh(\alpha-\gamma)+\frac{1}{2} \leq A$$ so that $$ |(\cosh(\alpha-\gamma)+\frac{1}{2})^{-1/4} - (\cosh \alpha + \frac{1}{2})^{-1/4}| \leq (e^{\gamma/4}-1) A^{-1/4},$$ so (again cancelling away the factors of $A$) it will suffice to show that $$ 2 \sigma e^{-\gamma/4} (\cosh \gamma - 1) \geq (e^{\gamma/4}-1)^2. \tag{(2)}$$ Comparing this with (1), we see that it will suffice to show (2) for all $\gamma \geq 0$ (since (1) then follows by discarding the $e^{-\gamma/4}$ factor on the left and then replacing $\gamma$ with $-\gamma$).
Writing $e^{\gamma/4} = s$, the inequality rearranges to $$ \sigma (s^4 + s^{-4}-2) - s (s-1)^2 \geq 0 \tag{(3)}$$ which can be numerically verified for $s \geq 1$:
We remark that this approach only barely works with the optimized value of $\sigma$; choosing a value of $\sigma$ that is even 1% smaller would not work, because (3) would now break down!
Alternative proof.
Remark. Fact 1 is inspired by Terry Tao's nice idea, that is, $\frac{\sqrt{3}}{2} \cos \frac{3(\pi-t)}{4} \leq \frac{3}{4} ( (\frac{1}{2}-\cos t) - \sigma (\frac{1}{2}-\cos t)^2 )$ in his answer.
Let $$A := a + \frac12, \quad B := b + \frac12, $$ $$p := \frac{\frac{1}{2} \left(a b -\sqrt{a^2-1} \sqrt{b^2-1}\right) - \frac12}{a + \frac12}, \quad q := \frac{\cos t - \frac12}{a + \frac12}. $$
The desired inequality is written as $$A^{3/4}(1 + p)^{3/4} -\frac{\sqrt{3}\, \cos\frac{3 (\pi -t)}{4}}{2 B^{1/4}} - A^{3/4}(1 + q)^{3/4} \ge 0. \tag{1}$$
By Bernoulli's inequality $(1 + u)^r \le 1 + ru$ for all $u > -1$ and $0 < r < 1$, it suffices to prove that $$A^{3/4}\cdot \frac{1 + p}{1 + p \cdot \frac{1}{4}} -\frac{\sqrt{3}\, \cos\frac{3 (\pi -t)}{4}}{2 B^{1/4}} - A^{3/4}\left(1 + \frac{3q}{4}\right) \ge 0$$ or (multiplying both sides by $4A^{1/4}$) $$4A\cdot \frac{1 + p}{1 + \frac{p}{4}} - 2\sqrt{3} \left(\frac{A}{B}\right)^{1/4} \cos\frac{3 (\pi -t)}{4} - 4A\left(1 + \frac{3q}{4}\right) \ge 0$$ or (using $Aq = \cos t - \frac12$) $$4A\cdot \frac{3p}{4 + p} - 2\sqrt{3} \left(\frac{A}{B}\right)^{1/4} \cos\frac{3 (\pi -t)}{4} - 3\left(\cos t - \frac12\right) \ge 0. \tag{2}$$
Note that $$ p = \frac{1}{2A}\cdot \frac{(a - b)^2}{(ab - 1) + \sqrt{a^2 - 1}\sqrt{b^2 - 1}} \ge \frac{1}{2A}\cdot \frac{(a - b)^2}{2ab}. $$ Thus, we have $$4A\cdot \frac{3p}{4 + p} \ge 4A\cdot \frac{3\cdot \frac{1}{2A}\cdot \frac{(a - b)^2}{2ab}}{4 + \frac{1}{2A}\cdot \frac{(a - b)^2}{2ab}} = \frac{6(2a + 1)(a - b)^2}{16a^2b + a^2 + 6ab + b^2}. \tag{3}$$
From (2) and (3), it suffices to prove that $$\frac{6(2a + 1)(a - b)^2}{16a^2b + a^2 + 6ab + b^2} - 2\sqrt{3} \left(\frac{a + \frac12}{b + \frac12}\right)^{1/4} \cos\frac{3 (\pi -t)}{4} - 3\left(\cos t - \frac12\right) \ge 0. \tag{4}$$
Fact 1. It holds that, for all $t\in [0, \pi]$, $$- 3\left(\cos t - \frac12\right) \ge 2\sqrt{3}\, \cos\frac{3 (\pi -t)}{4} + \frac12\left[\cos\frac{3 (\pi -t)}{4}\right]^2.$$ (Proof. Let $y = \cos \frac{\pi - t}{4}$. We have $\mathrm{LHS} - \mathrm{RHS} = \frac{-2y^3 - \sqrt{3}\, y^2 + 6y + \sqrt{3}}{2}(2y + \sqrt{3})(\sqrt{3} - 2y)^2 \ge 0$. )
By Fact 1, letting $u := \cos\frac{3 (\pi -t)}{4}$, it suffices to prove that $$\frac{6(2a + 1)(a - b)^2}{16a^2b + a^2 + 6ab + b^2} - 2\sqrt{3}\left[\left(\frac{a + \frac12}{b + \frac12}\right)^{1/4} - 1\right] u + \frac12 u^2\ge 0. \tag{5}$$
By AM-GM, it suffices to prove that $$4 \cdot \frac{6(2a + 1)(a - b)^2}{16a^2b + a^2 + 6ab + b^2} \cdot \frac12 \ge \left(2\sqrt{3}\left[\left(\frac{a + \frac12}{b + \frac12}\right)^{1/4} - 1\right]\right)^2. \tag{6}$$
It is easy to prove that, for all $x \ge 0$, $$(x^{1/4} - 1)^2 \le \frac{(x - 1)^2(1 + x)}{9x^2 + 22x + 1}. \tag{7}$$ From (7), letting $x = \left(\frac{a + \frac12}{b + \frac12}\right)^{1/4}$, we have $$\left[\left(\frac{a + \frac12}{b + \frac12}\right)^{1/4} - 1\right]^2 \le \frac{2(a + b + 1)(a - b)^2}{(2b + 1)(9a^2 + 22ab + b^2 + 20a + 12b + 8)}. \tag{8}$$
From (6) and (8), it suffices to prove that $$4 \cdot \frac{6(2a + 1)(a - b)^2}{16a^2b + a^2 + 6ab + b^2} \cdot \frac12 \ge 12 \cdot \frac{2(a + b + 1)(a - b)^2}{(2b + 1)(9a^2 + 22ab + b^2 + 20a + 12b + 8)}$$ or $$4 \cdot \frac{6(2a + 1)}{16a^2b + a^2 + 6ab + b^2} \cdot \frac12 \ge 12 \cdot \frac{2(a + b + 1)}{(2b + 1)(9a^2 + 22ab + b^2 + 20a + 12b + 8)}$$ or (after clearing the denominators) \begin{align*} &48\,{a}^{3}b+672\,{a}^{2}{b}^{2}+48\,a{b}^{3}+192\,{a}^{3}+1152\,{a}^{ 2}b+960\,a{b}^{2}\\ &\quad +564\,{a}^{2}+1272\,ba+276\,{b}^{2}+432\,a+336\,b+96\\ \ge{}& 0 \end{align*} which is true.
We are done.
