I had asked this question previously on Math.StacheExchange but did not get an answer there in several months. This isn't strictly speaking research level mathematics but I hope it is sufficiently advanced to tolerate it here. If that is not the case, I apologise. I use the physicists' convention where scalar products are linear in the second argument.
For complex numbers $z$ consider the system of $L^2(\mathbb{R})$-vectors with norm equal to $1$ \begin{equation} \psi_z=e^{-\frac{1}{2}|z|^2}\sum_{n=0}^{\infty}\frac{z^n}{\sqrt{n!}} |n\rangle, \end{equation} where $|n\rangle$ is an orthonormal basis of the hilbert space $L^2(\mathbb{R})$.
(Actually I'm only interested in the special case where the orthonormal basis is given by \begin{equation} |n\rangle (x)= \frac{\pi^{\frac{1}{4}}}{\sqrt{2^n n!}}H_n(x)\exp({-\frac{x^2}{2}}) \end{equation} with the Hermite-Polynomials $H_n$. This is the eigenbasis of the quantum harmonic oscillator with $m\omega =1=\hbar$. To add further background, the system $\psi_z$ are called coherent states and are closely linked to Segal-Bargmann spaces. I don't think the exact choice of basis matters though, in any case it doesn't seem to matter for calculations of the type needed to solve the problem below)
I'm trying to solve the following problem:
Let $P$ be a signed (real) measure on the Borel sets of $\mathbb{C}$, such that $P(\mathbb{C})=1.$ Consider the linear operator defined by $$R:L^2(\mathbb{R})\ni\phi\mapsto \int_{\mathbb{C}} \psi_z \langle \psi_z, \phi\rangle dP(z).$$ What are additional, sufficient conditions to the measure $P$, such that this operator becomes self-adjoint, positive and trace class with trace equal to $1$?
(This representation for a density operator is called "Glauber–Sudarshan P representation" and used in quantum optics) I am a physicist, and (no implication implied) my ability to make rigorous mathematical statements is somewhat lacking, especially when it comes to Bochner-integrals, as used in the definition of $R$ and, even worse, signed measures. Nevertheless I decided to get the math right on this one, which I need a little help for. Here is what I have done so far:
I'm already failing to prove that the operator is even well-defined as a bounded map. The main problem seems to be that any decomposition theorem for the measure will prevent me from using the fact that the measure of the whole plane is one. \begin{equation} \|R\phi\|\leq \int_{\mathbb{C}} |\langle \psi_z, \phi \rangle|\ d|P|(z)\leq \int_{\mathbb{C}} \| \phi\|\ d|P|=\|\phi\|\cdot\text{something finite?} \end{equation} using Cauchy-Schwarz on the second inequality. To get the first inequality, using the total variation $|P|$ of the measure, in a comment on the original question it was remaked by Eric Thoma, that one should assume $P$ to be decomposable into two sigma-finite positive measures, one of which (and thus also the other) is finite. Given this assumption I expect the last equality should also work out somehow. I haven't been able to prove this or find a source.
Next I would naively say that the operator is obviously self-adjoint, since $P$ is real. Also I would compute the trace of $R$ to be the reassuring \begin{align} \operatorname{tr} R=\sum_n \langle n, Rn\rangle&=\int_\mathbb{C}\sum_n \langle n,\psi_z\rangle \langle \psi_z , n\rangle dP(z)=\int_\mathbb{C}e^{-|z|^2}\sum_{n,m,k} \frac{z^m\overline{z}^k}{\sqrt{m!k!}}\langle n,m\rangle\langle k,n\rangle dP(z) \\ &= \int_\mathbb{C}1\ dP(z)=1. \end{align} However none of this is mathematically rigorous. What do I need to assume, so that my arguments work? I also don't know what exactly I need to assume in order for the resulting operator to be positive. This is the main point in question here, as the other problems are only about making rigorous sense of "physically clear" conditions. It seems the continuous part of the measure (w.r.t. Lebesgue) needs to be positive. That is a guess, I don't know if the condition is sufficient nor necessary.
Assuming that $P$ is decomposable into two sigma-finite positive measures, it seems reasonable to further assume that both the positive and negative parts should be finite on compact sets, and have vanishing "part" both absolutely continuous and singluar w.r.t. the Lebesgue measure on $\mathbb{C}$ in their unique decompositions. It is less of a problem to add too many conditions in this case, as I don't expect finding the sufficiant and necessary conditions will be doable. (I would also be very interested in finding conditions for replacing integration against the measure by the action of certain (probably tempered) distributions, that seems even harder though) Thank you all very much in advance for the help!
