MathOverflow will be down for maintenance for approximately 3 hours, starting Monday evening (06/24/2013) at approximately 9:00 PM Eastern time (UTC-4).

Write your quantity as: $$f(A)=\hbox{Tr}\left[ P_A|0\rangle\langle0|P_A(\mathbb{1}-|0\rangle\langle0|) \right]$$ $$= \hbox{Tr}[P_A|0\rangle\langle0|]-\hbox{Tr}[(P_A|0\rangle\langle0|)^2],$$ right],$$where P_A is the projection on A, and |0\rangle\langle0| is the projection on \varphi_0. Note that then you need only to investigate properties of the \varphi_0, not every Hermite function (as they form an orthonormal basis). With the properties of Tr and projection operators you get$$f(A)= \hbox{Tr}[|0\rangle\langle0|P_A|0\rangle\langle0|]-\hbox{Tr}[(|0\rangle\langle0|P_A|0\rangle\langle0|)^2]=\lambda-\lambda^2.$$So: • indeed, \max f(A) = \frac{1}{4}, • f(A)=\frac{1}{4} iff \int_{A} \varphi_0^2(x)dx=\frac{1}{2}. Depending what you need the formula for, but if it is about the filtering of the higher-order modes, here (sec. 6.) is a numerical remark. 2 added 13 characters in body I don't know answer, but the following might be useful. Write your quantity as:$$f(A)=\hbox{Tr}\left[ P_A|0\rangle\langle0|P_A(\mathbb{1}-|0\rangle\langle0|) \right]= \hbox{Tr}[P_A|0\rangle\langle0|]-\hbox{Tr}[(P_A|0\rangle\langle0|)^2],$$where P_A is the projection on A, and |0\rangle\langle0| is the projection on \varphi_0. Note that then you need only to investigate properties of the \varphi_0, not every Hermite function (as they form a an orthonormal basis). Depending what you need the formula for, but if it is about the filtering of the higher-order modes, here (sec. 6.) is a numerical remark. 1 I don't know answer, but the following might be useful. Write your quantity as:$$f(A)=\hbox{Tr}\left[ P_A|0\rangle\langle0|P_A(\mathbb{1}-|0\rangle\langle0|) \right]= \hbox{Tr}[P_A|0\rangle\langle0|]-\hbox{Tr}[(P_A|0\rangle\langle0|)^2], where $P_A$ is the projection on A, and $|0\rangle\langle0|$ is the projection on $\varphi_0$. Note that then you need only to investigate properties of the $\varphi_0$, not every Hermite function (as they form a basis).