We will perform computation using
normed barycentric coordinates in the given triangle $\Delta ABC$ with side lengths $a,b,c$ and angles
$\hat A=3\alpha$,
$\hat B=3\beta$,
$\hat C=3\gamma$.
We denote by $R$ the circumradius of the circle $\odot(ABC)$, and the area $[ABC]$ by $S$.
Trilinear coordinates of the points $A'$, $B'$, $C'$ (vertices of the first Morley triangle of
$\Delta ABC$) can be extracted from the rows of the matrix:
$$
\begin{bmatrix}
1 & 2\cos \gamma & 2\cos\beta\\
2\cos \gamma & 1 & 2\cos\alpha\\
2\cos \beta & 2\cos\alpha & 1\\
\end{bmatrix}
\ .
$$
(See for instance https://mathworld.wolfram.com/FirstMorleyTriangle.html .)
The (notnormalized) barycentric coordinates can then be extracted from the rows of the matrix:
$$
\tag{$*$}
\begin{bmatrix}
a & 2b\cos \gamma & 2c\cos\beta\\
2a\cos \gamma & b & 2c\cos\alpha\\
2a\cos \beta & 2c\cos\alpha & c\\
\end{bmatrix}
\ .
$$
Let now $P_A$, $P_B$, $P_C$ be three points with normalized barycentric coordinates
$$
\begin{aligned}
P_A &= (x_A, y_A, z_A)\ ,\\
P_B &= (x_B, y_B, z_B)\ ,\\
P_C &= (x_C, y_C, z_C)\ .
\end{aligned}
$$
Define $Q_A$ to be such that $Q_AP_B\perp AC$ and $Q_AP_C\perp AB$.
Construct in a similar manner $Q_B,Q_C$.
Computer algebra shows that
$$
\begin{aligned}
s^2 :=\ &\frac{[Q_AQ_BQ_C]}{[ABC]}
=\frac{P_BP_C}{a^2}
=\frac{P_CP_A}{b^2}
=\frac{P_AP_B}{c^2}
\qquad\text{ is given by the relation}
\\
16S^2\cdot s^2 =\
&\
\begin{pmatrix}
+ x_A(b^2c^2) + a^2(y_Az_A) \\
+ y_B(c^2a^2) + b^2(z_Bx_B) \\
+ z_C(a^2b^2) + c^2(x_Cy_C)
\end{pmatrix}
^2 \ .
\end{aligned}
$$
(Keeping $A$ and exchanging $B\leftrightarrow C$ moves $s$ to $s$.)
So we have to maximize the expression
$$
\begin{aligned}
s
&= \frac 1{4S}
\begin{pmatrix}
+ x_A(b^2c^2) + a^2(y_Az_A) \\
+ y_B(c^2a^2) + b^2(z_Bx_B) \\
+ z_C(a^2b^2) + c^2(x_Cy_C)
\end{pmatrix}
\\
&=\frac 1{4S}\sum x_A(b^2c^2) + a^2(y_Az_A)
\\
&=\frac 1{4S}\sum (1y_Az_A)(b^2c^2) + a^2(y_Az_A)
\\
&=\frac 1{4S}\sum y_A(a^2b^2+c^2) z_A(a^2+b^2c^2)
\\
&=\frac 1{4S}\sum y_A\cdot 2ac\sin B z_A\cdot 2ab\sin C
\\
&=\frac 1{4RS}\sum y_A\cdot abc z_A\cdot abc
\\
&=\sum(y_A  z_A)\ .
\\[3mm]
&\qquad\text{Now we plug in the values for $y_A$, $z_A$ from $(*)$:}
\\[3mm]
y_A &=
\frac
{2\cdot 2R\sin B\cos\gamma}
{2R(\sin A +2\cdot \sin B\cos \gamma +2\cdot \sin C\cos \beta)}\ ,
\\
z_A &=
\frac
{2\cdot 2R\sin C\cos\beta}
{2R(\sin A +2\cdot \sin B\cos \gamma +2\cdot \sin C\cos \beta)}\ ,
\\[3mm]
&\qquad\text{getting:}
\\[3mm]
s
&=\sum
\frac
{2\cdot \sin B\cos \gamma  2\sin C\cos\beta}
{\sin A +2\cdot \sin B\cos \gamma +2\cdot \sin C\cos \beta}
% \\
% &=3 + \sum
% \frac
% {\sin A + 2\cdot \sin B\cos \gamma}
% {\sin A + 2\cdot \sin B\cos \gamma + 2\cdot \sin C\cos \beta}
\ .
\end{aligned}
$$
This is not an easy task now. (Either for the last expression of $s$,
or for the formula in between with a cyclic sum $s=\sum(y_Az_A)$, if there is some
better geometric interpretation.)
So we are starting to solve a new problem in the problem.
Since time is an issue for me, instead of starting to compute using Lagrange multipliers,
in order to proceed in some few lines
here is a picture of the function to be maximized:
The plot uses $x,y\in[0,1]$ to parametrize the angles $\alpha,\beta,\gamma$ as follows:
$\displaystyle \alpha = \frac\pi3\cdot x$,
$\displaystyle \beta = \frac\pi3\cdot y(1x)$, and
$\displaystyle \gamma = \frac\pi3\cdot (1y)(1x)$.
It turns out that the maximum is in fact a supremum, taken for the case when
$\displaystyle \alpha\nearrow\frac \pi 3$,
and then $\displaystyle \beta,\gamma\searrow 0$, so that
$\displaystyle \alpha+\beta+\gamma=\frac \pi 3$.
In terms of the angles of $\Delta ABC$ we have $\hat A\nearrow\pi$, $\hat B,\hat C\searrow 0$. (But $\hat B$, $\hat C$ may need to be still correlated.)
To compute this supremum, i need a notation that i can better type.
So i will switch from $\alpha,\beta,\gamma$ to $x,y,z$.
Then $\hat B=3y$, $\hat C=3z$, and corresponding sine values
will be approximated by hand waving with $3y+O(y^3)$ and $3z+O(z^3)$,
so that for $\hat A=\pi(3x+3y)$ we have also a sine value in the shape $3y+3z+O(\dots)$.
Computations will omit below terms in total monomial degree bigger / equal two.
So
$$
\begin{aligned}
s &=
\frac
{2\sin B\cos\frac C3  2\sin C\cos \frac B3}
{\sin A + 2\sin B\cos\frac C3 + 2\sin C\cos \frac B3}
\\
&\qquad
+\frac
{2\sin C\cos\frac A3  2\sin A\cos \frac C3}
{\sin B + 2\sin C\cos\frac A3 + 2\sin A\cos \frac C3}
\\
&\qquad\qquad
+\frac
{2\sin A\cos\frac B3  2\sin B\cos \frac A3}
{\sin C + 2\sin A\cos\frac B3 + 2\sin B\cos \frac A3}
\\
&\sim
\frac{2(3y3z)}{3y+3z + 2(3y+3z)}\\
&\qquad
+\frac {2\left(3z\cdot\frac 12 (3y+3z)\right)}{3y + 2\left(3z\cdot\frac 12 +(3y+3z)\right)}\\
&\qquad\qquad
+\frac {2\left((3y+3z)  3y\cdot\frac 12\right)}{3z + 2\left((3x+3y) + 3y\cdot\frac 12\right)}
\\
&=
\frac {2y2z}{3y+3z}

\frac {2y+z}{3y+3z}
+
\frac {y+2z}{3y+3z}
\\
&=\frac{yz}{3y+3z}\ .
\end{aligned}
$$
So we expect $\displaystyle\color{blue}{\frac 13}$ as a supremum value.