I don't know answer, but the following might be useful.
Write your quantity as: $$f(A)=\hbox{Tr}\left[ P_A|0\rangle\langle0|P_A(\mathbb{1}-|0\rangle\langle0|) \right]$$ $$= \hbox{Tr}[P_A|0\rangle\langle0|]-\hbox{Tr}[(P_A|0\rangle\langle0|)^2],$$ right],$$ where $P_A$ is the projection on A, and $|0\rangle\langle0|$ is the projection on $\varphi_0$. Note that then you need only to investigate properties of the $\varphi_0$, not every Hermite function (as they form an orthonormal basis).
With the properties of Tr and projection operators you get $$f(A)= \hbox{Tr}[|0\rangle\langle0|P_A|0\rangle\langle0|]-\hbox{Tr}[(|0\rangle\langle0|P_A|0\rangle\langle0|)^2]$$ $$=\lambda-\lambda^2.$$
So:
- indeed, $\max f(A) = \frac{1}{4}$,
- $f(A)=\frac{1}{4}$ iff $\int_{A} \varphi_0^2(x)dx=\frac{1}{2}$.
Depending what you need the formula for, but if it is about the filtering of the higher-order modes, here (sec. 6.) is a numerical remark.
