"psi-epistemic theories" in 3 or more dimensions

Question

In their recent paper The Quantum State Can Be Interpreted Statistically, Lewis et al. end with a very nice mathematical question, one whose answer (either way) would have interesting implications for the foundations of quantum mechanics. In the hopes of getting some non-quantum math folks interested in their question---and maybe even finding someone to say "the answer is trivial for the following reason..." :-)---I decided to state the question for the MO community, shorn of all the physics and philosophy.

Let H_d be the set of unit vectors in $\mathbb{C}^d$. A ψ-epistemic theory in d dimensions consists of the following:

1. A measurable space Λ (called the "space of ontic states").
2. A function mapping each unit vector ψ∈H_d to a probability measure D_ψ over Λ.
3. A function f(λ,M,i)∈[0,1], which takes as input an ontic state λ∈Λ, an ordered orthonormal basis M=(v₁,...,v_d) for $\mathbb{C}^d$, and an index i∈{1,...,d}.

f must satisfy the following two conditions:

(i) $\sum_{i=1}^{d}f(\lambda,M,i)=1$ for all λ and M. (Intuitively, f must give rise to a probability distribution over the "measurement outcomes" v₁,...,v_d in M.)

(ii) $\int_{\lambda \sim D_{\psi}} f(\lambda,M,i) d\lambda = |v_{i}^{*}\psi|^{2}$ for all ψ, M, and i. (Intuitively, the probability of the measurement outcome v_i, averaged over all λ drawn from D_ψ, must equal the squared projection of ψ onto v_i.)

Note that we can trivially satisfy conditions (i) and (ii) as follows:

• Λ=H_d
• D_ψ assigns probability 1 to λ=ψ, and probability 0 to all other states in Λ
• f(ψ,M,i) = |v_i^*ψ|²

Thus, let Supp(D)⊆Λ be the support of D, and call a ψ-epistemic theory nontrivial if there exist ψ≠ϕ such that $Supp(D_{\psi})\cap Supp(D_{\phi}) \ne \emptyset$.

Observe that, if ψ and ϕ are orthogonal, then Supp(D_ψ) and Supp(D_ϕ) must be disjoint. This is because, if we set v₁=ψ and v₂=ϕ, then $v_{1}^{*}\psi = v_{2}^{*}\phi = 1$ and $v_{1}^{*}\phi = v_{2}^{*}\psi = 0$, which is not possible if D_ψ and D_ϕ have any nonzero overlap. Motivated by this observation, call a theory maximally nontrivial if $Supp(D_{\psi})\cap Supp(D_{\phi}) \ne \emptyset$ whenever ψ and ϕ are not orthogonal.

I can now state Lewis et al.'s open problem:

Does there exist a maximally-nontrivial ψ-epistemic theory in dimensions d≥3?

Update: See the comments for an extremely nice solution by George Lowther, plus my followup questions.

I know of two results directly relevant to this problem.

First, there exists a maximally-nontrivial theory in dimension d=2, which was found by Kochen and Specker in 1967. See this paper by Rudolph for more details, including why the obvious generalizations to 3 or more dimensions seem to fail. Briefly, the Kochen-Specker theory is defined as follows:

• Λ=H₂.
• D_ψ assigns probability measure $2 | \psi^{\*} \phi|^{2} - 1$ to ϕ if $| \psi^{\*} \phi|^{2} \geq 1/2$, and probability measure 0 to ϕ otherwise.
• f(ψ,M,i) = 1 if $|v_{i}^{\*} \psi|^{2} \geq 1/2$, and f(ψ,M,i) = 0 otherwise.

(Warning: I converted from a different representation, and can't promise I didn't get a factor of 2 wrong or something like that.)

The second result is that, for all finite d, there exists a nontrivial ψ-epistemic theory (though it's far from being maximally nontrivial). This is the main result of Lewis et al.

My own guess is that maximally-nontrivial theories don't exist for d≥3, but I'd only give it 60% confidence.

To anticipate some questions:

• Yes, I'd also be interested in this problem with $\mathbb{R}$ in place of $\mathbb{C}$ (though I suspect the two cases are pretty similar).

• Yes, I'd be interested in negative results for restricted classes of theories. Here are a few examples of restrictions one could look at, in various combinations: Λ=H_d, f∈{0,1}, f is continuous, symmetry under unitary transformations, symmetry under relabeling of the v_i's.

• No, I don't know how to rule out that the answer could depend on the Axiom of Choice or something crazy like that (but I doubt it).

Update (March 20, 2013): Adam Bouland, Lynn Chua, George Lowther, and myself now have a paper on ψ-epistemic theories originating with this MO post. The paper contains the construction below, but also proves impossibility results for ψ-epistemic theories when an additional symmetry condition is imposed.

Answer 1

Since George Lowther seems to have a lot of late nights, I decided to express my gratitude to him by writing up his lovely answer myself and thereby saving him the trouble.

The answer to my (and Lewis et al.'s) question is that yes, maximally-nontrivial ψ-epistemic theories do exist for every finite dimension $d$.

The first realization is that we can "mix" small ε-balls around any two non-orthogonal vectors.

Lemma 1: Given any two non-orthogonal unit vectors $\psi,\phi \in \mathbb{C}^d$, there exists a ψ-epistemic theory $T = T(\psi,\phi)$ such that $Supp(D_{\psi})$ and $Supp(D_{\phi})$ have nonempty intersection. Moreover, for this $T$, there exists an $\epsilon \gt 0$ (for example, $\epsilon = |\psi^{*} \phi|/2d$) such that $Supp(D_{\psi'})$ and $Supp(D_{\phi'})$ have nonempty intersection for all $\psi',\phi'$ such that $||\psi - \psi'||,||\phi - \phi'||\lt \epsilon$.

Proof: Our ontic state space will be $\Lambda = H_{d} \times [0,1]$. Given an orthonormal basis $M=(v_1,...,v_d)$, first sort the $v_i$'s in decreasing order of $min(|v_{i}^{∗}\psi|,|v_{i}^{∗}\phi|)$. Then the outcome of measurement M on ontic state $(w,p)\in \Lambda$ will equal the smallest positive integer $i$ such that

$|v_1^* w|^2+...+|v_{i−1}^*w|^2 \le p \le |v_1^* w|^2+...+|v_i^*w|^2$.

In other words, $f((w,p),M,i)$ will equal $1$ if $i$ satisfies the above and no $j \lt i$ does, and $0$ otherwise. For now, assume that $D_{w}$ is simply the uniform distribution over all states $(w,p)$ with $p \in [0,1]$. It is easy to check that this yields a valid ψ-epistemic theory, albeit so far a trivial one.

Since $|\psi^{*} \phi|\gt 0$, I claim that there exists an $\epsilon \gt 0$ such that for all orthonormal bases $M=(v_1,...,v_d)$, there exists an $i$ such that $|v_{i}^{∗}\psi|\ge \epsilon$ and $|v_{i}^{∗}\phi|\ge \epsilon$. Indeed, by the triangle inequality, setting $\epsilon := |\psi^{*} \phi| / d$ will work.

Now, the above means that, for all measurements $M$ and all $p \in [0,\epsilon]$, the outcome is always $i=1$ when $M$ is applied to either of the ontic states $(\psi,p)$ or $(\phi,p)$. Following Lewis et al., this implies that we can "mix" the corresponding distributions $D_{\psi}$ and $D_{\phi}$―i.e., have them intersect each other in the region $p \in [0,\epsilon]$―without affecting any outcome of any measurement $M$.

Furthermore, suppose $\psi'$ and $\phi'$ are $\epsilon /2$-close to $\psi$ and $\phi$ respectively, in some standard metric such as trace distance. Then by continuity, we can similarly mix the distributions $D_{\psi'}$ and $D_{\phi'}$―i.e., have them intersect each other in the region $p\in [0,\epsilon /2]$―without affecting any measurement outcome. (One subtlety is that, as we vary $M$, the sorting procedure can make $v_1$ "jump" discontinuously from one basis vector of $M$ to another. However, this jumping is not a problem, since it depends only on the fixed vectors $\psi$ and $\phi$, not on $\psi'$ or $\phi'$. So it happens the same way everywhere in the $\epsilon /2$-balls.) QED

The second realization is that we can take "convex combinations" of ψ-epistemic theories. Given two ψ-epistemic theories $T=(\Lambda,D,f)$ and $T'=(\Lambda',D',f')$ (where $D,D'$ are the functions that map vectors $\psi \in H^d$ onto ontic distributions), and a constant $c \in (0,1)$, define the new theory $c T + (1-c)T' =(\Lambda_c,D_c,f_c)$ as follows:

• $\Lambda_c := \Lambda \cup \Lambda'$.
• $D_c := c D + (1-c) D'$.
• $f_c : \Lambda_c \rightarrow [0,1]$ equals $f$ on $\Lambda$ and $f'$ on $\Lambda'$.

Lemma 2: $c T + (1-c)T'$ is a ψ-epistemic theory. Furthermore, if $T$ mixes the ontic distributions of two vectors $\psi$ and $\phi$, and $T'$ mixes the ontic distributions of two other vectors $\psi'$ and $\phi'$, then $c T + (1-c)T'$ mixes both pairs of distributions.

Proof: Immediate.

Using Lemmas 1 and 2, we now construct a maximally-nontrivial ψ-epistemic theory. Let $T(\psi,\phi)$ be the theory returned by Lemma 1 given vectors $\psi,\phi\in H^d$. Also, for all positive integers $n$, let $A_n$ be a $1/n$-net for $H^d$: that is, a finite subset $A_n \subseteq H^d$ such that for all $v\in H^d$, there exists a $w \in A_n$ satisfying $|| w - v || \lt 1/n$. By making small perturbations, we can easily ensure the property that $u^{*}v \neq 0$ for all $u,v\in A_n$. Then our theory $T$ is defined as follows:

$$T = \frac{6}{\pi^2} \sum_{n=1}^{\infty} \frac{1}{n^2} \left( \frac{1}{|A_n|^2} \sum_{u,v \in A_n} T(u,v) \right). $$

One can check that $T$ mixes the distributions $D_{\psi}$ and $D_{\phi}$ for all non-orthogonal $\psi$ and $\phi$.

Comment: As often in math, I'd say the true value of knowing this answer is that it points us toward the questions we (or rather Lewis et al.) "really" meant to ask! In the above construction, the overlap between $D_{\psi}$ and $D_{\phi}$ is indeed nonzero for any non-orthogonal $\psi,\phi$, but the amount of overlap falls off (by my crude estimate) as $ ( |\psi^{*} \phi| / d) ^{ \Theta(d) } $ .

A skeptic of ψ-epistemic theories might argue that for large $d$ (and of course, $d$ can be huge in quantum mechanics), such an overlap is physically irrelevant. So one obvious followup question is how large the overlap can be―for example, whether it can fall off only as $(|\psi^{*} \phi| / d)^{O(1)}$. I'd better stop here, though, since I know MO is not for open-ended research discussions. The question, as I stated it, has been answered.