# “psi-epistemic theories” in 3 or more dimensions

In their recent paper The Quantum State Can Be Interpreted Statistically, Lewis et al. end with a very nice mathematical question, one whose answer (either way) would have interesting implications for the foundations of quantum mechanics. In the hopes of getting some non-quantum math folks interested in their question---and maybe even finding someone to say "the answer is trivial for the following reason..." :-)---I decided to state the question for the MO community, shorn of all the physics and philosophy.

Let Hd be the set of unit vectors in $\mathbb{C}^d$. A ψ-epistemic theory in d dimensions consists of the following:

1. A measurable space Λ (called the "space of ontic states").
2. A function mapping each unit vector ψ∈Hd to a probability measure Dψ over Λ.
3. A function f(λ,M,i)∈[0,1], which takes as input an ontic state λ∈Λ, an ordered orthonormal basis M=(v1,...,vd) for $\mathbb{C}^d$, and an index i∈{1,...,d}.

f must satisfy the following two conditions:

(i) $\sum_{i=1}^{d}f(\lambda,M,i)=1$ for all λ and M. (Intuitively, f must give rise to a probability distribution over the "measurement outcomes" v1,...,vd in M.)

(ii) $\int_{\lambda \sim D_{\psi}} f(\lambda,M,i) d\lambda = |v_{i}^{*}\psi|^{2}$ for all ψ, M, and i. (Intuitively, the probability of the measurement outcome vi, averaged over all λ drawn from Dψ, must equal the squared projection of ψ onto vi.)

Note that we can trivially satisfy conditions (i) and (ii) as follows:

• Λ=Hd
• Dψ assigns probability 1 to λ=ψ, and probability 0 to all other states in Λ
• f(ψ,M,i) = |vi*ψ|2

Thus, let Supp(D)⊆Λ be the support of D, and call a ψ-epistemic theory nontrivial if there exist ψ≠ϕ such that $Supp(D_{\psi})\cap Supp(D_{\phi}) \ne \emptyset$.

Observe that, if ψ and ϕ are orthogonal, then Supp(Dψ) and Supp(Dϕ) must be disjoint. This is because, if we set v1=ψ and v2=ϕ, then $v_{1}^{*}\psi = v_{2}^{*}\phi = 1$ and $v_{1}^{*}\phi = v_{2}^{*}\psi = 0$, which is not possible if Dψ and Dϕ have any nonzero overlap. Motivated by this observation, call a theory maximally nontrivial if $Supp(D_{\psi})\cap Supp(D_{\phi}) \ne \emptyset$ whenever ψ and ϕ are not orthogonal.

I can now state Lewis et al.'s open problem:

Does there exist a maximally-nontrivial ψ-epistemic theory in dimensions d≥3?

Update: See the comments for an extremely nice solution by George Lowther, plus my followup questions.

I know of two results directly relevant to this problem.

First, there exists a maximally-nontrivial theory in dimension d=2, which was found by Kochen and Specker in 1967. See this paper by Rudolph for more details, including why the obvious generalizations to 3 or more dimensions seem to fail. Briefly, the Kochen-Specker theory is defined as follows:

• Λ=H2.
• Dψ assigns probability measure $2 | \psi^{\*} \phi|^{2} - 1$ to ϕ if $| \psi^{\*} \phi|^{2} \geq 1/2$, and probability measure 0 to ϕ otherwise.
• f(ψ,M,i) = 1 if $|v_{i}^{\*} \psi|^{2} \geq 1/2$, and f(ψ,M,i) = 0 otherwise.

(Warning: I converted from a different representation, and can't promise I didn't get a factor of 2 wrong or something like that.)

The second result is that, for all finite d, there exists a nontrivial ψ-epistemic theory (though it's far from being maximally nontrivial). This is the main result of Lewis et al.

My own guess is that maximally-nontrivial theories don't exist for d≥3, but I'd only give it 60% confidence.

To anticipate some questions:

• Yes, I'd also be interested in this problem with $\mathbb{R}$ in place of $\mathbb{C}$ (though I suspect the two cases are pretty similar).

• Yes, I'd be interested in negative results for restricted classes of theories. Here are a few examples of restrictions one could look at, in various combinations: Λ=Hd, f∈{0,1}, f is continuous, symmetry under unitary transformations, symmetry under relabeling of the vi's.

• No, I don't know how to rule out that the answer could depend on the Axiom of Choice or something crazy like that (but I doubt it).

Update (March 20, 2013): Adam Bouland, Lynn Chua, George Lowther, and myself now have a paper on ψ-epistemic theories originating with this MO post. The paper contains the construction below, but also proves impossibility results for ψ-epistemic theories when an additional symmetry condition is imposed.

-
Maximally nontrivial theories must exist for any finite $d$. For a nontrivial $\psi$-epistemic theory with given $\phi,\psi$ you only need to verify that $p=\inf_M\max_i min(\vert\nu_i^*\phi\vert,\vert\nu_i^*\psi\vert) > 0$. This holds by compactness+continuity so that the extrema are obtained, and must also hold in some neighborhood of $\phi,\psi$. You can boost this up by taking convex combinations of such theories and using compactess to cover all possible $\phi,\psi$. – George Lowther Apr 30 '12 at 23:41
(I think that should answer the question, but Its getting a bit late right now to be writing up a full answer). Also, you can turn any theory with $f\in[0,1]$ to one with $f\in\lbrace0,1\rbrace$ quite easily, so that is no restriction. – George Lowther Apr 30 '12 at 23:48
In fact, it isn't particularly dependent on the rules of QM. It just relies on the following facts. (i) The space M of measurements and the space S of states are compact. (ii) The map from measurement x state to probability distribution of the measurement result is continuous. (iii) for any pair of "non-orthogonal" states and any measurement, the pair of probability distributions for the measurement result are non-singular. – George Lowther May 1 '12 at 1:25
Thanks, George!! I follow your very nice solution modulo one step. Lewis et al.'s nontrivial ψ-epistemic theory can only mix $D_ϕ$ and $D_ψ$ if $ψ^{∗}ϕ$ is sufficiently close to 1. So to finish your argument, I think we need one additional lemma. Namely: there exists a fixed ontic state space Λ, such that for all non-orthogonal ψ,ϕ, there exists a ψ-epistemic theory T mapping onto Λ, as well as an $\epsilon\gt 0$, such that T mixes $D_{ψ′}$ with $D_{ϕ′}$ for all ψ',ϕ' in $\epsilon$-neighborhoods of ψ,ϕ respectively. – Scott Aaronson May 1 '12 at 2:53
That lemma seems plausible; let me try to prove it. Assuming such T's exist, your idea of taking a convex combination of theories should indeed work (of course, we'll need an infinite convex combination, to handle the limit as $|ψ^{∗}ϕ|$ approaches 0, since clearly we'll always have $\epsilon \lt |ψ^{∗}ϕ|$). – Scott Aaronson May 1 '12 at 3:00

Since George Lowther seems to have a lot of late nights, I decided to express my gratitude to him by writing up his lovely answer myself and thereby saving him the trouble.

The answer to my (and Lewis et al.'s) question is that yes, maximally-nontrivial ψ-epistemic theories do exist for every finite dimension $d$.

The first realization is that we can "mix" small ε-balls around any two non-orthogonal vectors.

Lemma 1: Given any two non-orthogonal unit vectors $\psi,\phi \in \mathbb{C}^d$, there exists a ψ-epistemic theory $T = T(\psi,\phi)$ such that $Supp(D_{\psi})$ and $Supp(D_{\phi})$ have nonempty intersection. Moreover, for this $T$, there exists an $\epsilon \gt 0$ (for example, $\epsilon = |\psi^{*} \phi|/2d$) such that $Supp(D_{\psi'})$ and $Supp(D_{\phi'})$ have nonempty intersection for all $\psi',\phi'$ such that $||\psi - \psi'||,||\phi - \phi'||\lt \epsilon$.

Proof: Our ontic state space will be $\Lambda = H_{d} \times [0,1]$. Given an orthonormal basis $M=(v_1,...,v_d)$, first sort the $v_i$'s in decreasing order of $min(|v_{i}^{∗}\psi|,|v_{i}^{∗}\phi|)$. Then the outcome of measurement M on ontic state $(w,p)\in \Lambda$ will equal the smallest positive integer $i$ such that

$|v_1^* w|^2+...+|v_{i−1}^*w|^2 \le p \le |v_1^* w|^2+...+|v_i^*w|^2$.

In other words, $f((w,p),M,i)$ will equal $1$ if $i$ satisfies the above and no $j \lt i$ does, and $0$ otherwise. For now, assume that $D_{w}$ is simply the uniform distribution over all states $(w,p)$ with $p \in [0,1]$. It is easy to check that this yields a valid ψ-epistemic theory, albeit so far a trivial one.

Since $|\psi^{*} \phi|\gt 0$, I claim that there exists an $\epsilon \gt 0$ such that for all orthonormal bases $M=(v_1,...,v_d)$, there exists an $i$ such that $|v_{i}^{∗}\psi|\ge \epsilon$ and $|v_{i}^{∗}\phi|\ge \epsilon$. Indeed, by the triangle inequality, setting $\epsilon := |\psi^{*} \phi| / d$ will work.

Now, the above means that, for all measurements $M$ and all $p \in [0,\epsilon]$, the outcome is always $i=1$ when $M$ is applied to either of the ontic states $(\psi,p)$ or $(\phi,p)$. Following Lewis et al., this implies that we can "mix" the corresponding distributions $D_{\psi}$ and $D_{\phi}$―i.e., have them intersect each other in the region $p \in [0,\epsilon]$―without affecting any outcome of any measurement $M$.

Furthermore, suppose $\psi'$ and $\phi'$ are $\epsilon /2$-close to $\psi$ and $\phi$ respectively, in some standard metric such as trace distance. Then by continuity, we can similarly mix the distributions $D_{\psi'}$ and $D_{\phi'}$―i.e., have them intersect each other in the region $p\in [0,\epsilon /2]$―without affecting any measurement outcome. (One subtlety is that, as we vary $M$, the sorting procedure can make $v_1$ "jump" discontinuously from one basis vector of $M$ to another. However, this jumping is not a problem, since it depends only on the fixed vectors $\psi$ and $\phi$, not on $\psi'$ or $\phi'$. So it happens the same way everywhere in the $\epsilon /2$-balls.) QED

The second realization is that we can take "convex combinations" of ψ-epistemic theories. Given two ψ-epistemic theories $T=(\Lambda,D,f)$ and $T'=(\Lambda',D',f')$ (where $D,D'$ are the functions that map vectors $\psi \in H^d$ onto ontic distributions), and a constant $c \in (0,1)$, define the new theory $c T + (1-c)T' =(\Lambda_c,D_c,f_c)$ as follows:

• $\Lambda_c := \Lambda \cup \Lambda'$.
• $D_c := c D + (1-c) D'$.
• $f_c : \Lambda_c \rightarrow [0,1]$ equals $f$ on $\Lambda$ and $f'$ on $\Lambda'$.

Lemma 2: $c T + (1-c)T'$ is a ψ-epistemic theory. Furthermore, if $T$ mixes the ontic distributions of two vectors $\psi$ and $\phi$, and $T'$ mixes the ontic distributions of two other vectors $\psi'$ and $\phi'$, then $c T + (1-c)T'$ mixes both pairs of distributions.

Proof: Immediate.

Using Lemmas 1 and 2, we now construct a maximally-nontrivial ψ-epistemic theory. Let $T(\psi,\phi)$ be the theory returned by Lemma 1 given vectors $\psi,\phi\in H^d$. Also, for all positive integers $n$, let $A_n$ be a $1/n$-net for $H^d$: that is, a finite subset $A_n \subseteq H^d$ such that for all $v\in H^d$, there exists a $w \in A_n$ satisfying $|| w - v || \lt 1/n$. By making small perturbations, we can easily ensure the property that $u^{*}v \neq 0$ for all $u,v\in A_n$. Then our theory $T$ is defined as follows:

$$T = \frac{6}{\pi^2} \sum_{n=1}^{\infty} \frac{1}{n^2} \left( \frac{1}{|A_n|^2} \sum_{u,v \in A_n} T(u,v) \right).$$

One can check that $T$ mixes the distributions $D_{\psi}$ and $D_{\phi}$ for all non-orthogonal $\psi$ and $\phi$.

Comment: As often in math, I'd say the true value of knowing this answer is that it points us toward the questions we (or rather Lewis et al.) "really" meant to ask! In the above construction, the overlap between $D_{\psi}$ and $D_{\phi}$ is indeed nonzero for any non-orthogonal $\psi,\phi$, but the amount of overlap falls off (by my crude estimate) as $( |\psi^{*} \phi| / d) ^{ \Theta(d) }$ .

A skeptic of ψ-epistemic theories might argue that for large $d$ (and of course, $d$ can be huge in quantum mechanics), such an overlap is physically irrelevant. So one obvious followup question is how large the overlap can be―for example, whether it can fall off only as $(|\psi^{*} \phi| / d)^{O(1)}$. I'd better stop here, though, since I know MO is not for open-ended research discussions. The question, as I stated it, has been answered.

-
Thanks for that. I might still post my own answer using the slightly more abstract and general argument in my 3rd comment. Also, I think I'd count myself among the "skeptics of $\psi$-epistemic theories"! – George Lowther May 2 '12 at 22:38
Yeah, I'd consider myself a thoroughgoing skeptic of these theories too, as "descriptions of what's really going on" ... I just found the mathematical questions about them unexpectedly interesting! There's some more discussion about this over at my blog (scroll down to the comments section): scottaaronson.com/blog/?p=993 – Scott Aaronson May 2 '12 at 22:49
Nice construction. However this kind of ψ-epistemic theory has an ugly property. Given a quantum state ψ, there is finite probability that the generated ontic state contains the full information about the quantum state. In this sense, this model is partially $\psi$-ontic. So, I have a question. Is it possible to construct a full ψ-epistemic theory where the ontic state never contains the full information about the quantum state? – Alberto Montina May 12 '12 at 21:32
Manas Patra made a comment about your construction of convex combination of theories in a deleted answer. He points out that you make an implicit assumption that $\Lambda$ and $\Lambda'$ are disjoint, since otherwise $f_c$ may not be well-defined. He or she seems to prefer Lowther's original suggestion that the two theories define the same ontic space, so that one takes the convex combination of f's as well – S. Carnahan Feb 4 '13 at 6:59
Sorry for the delay! Manas Patra and S. Carnahan: I also thought about defining convex combinations of theories, $T_c=cT+(1-c)T'$, by using the same ontic space, and letting $f_c=cf+(1-c)f'$. But I think there's a problem: the resulting theory $T_c$ might no longer reproduce quantum mechanics, even if $T$ and $T'$ did. If we were only mixing $f$'s or only mixing $D$'s, then we'd be fine, by linearity. But when we mix both of them, we get unwanted cross-terms: the integral of $f(\lambda,M,i)d \lambda$ over $\lambda$ drawn from $D'$, and of $f'(\lambda,M,i)d \lambda$ over $\lambda$ from $D$. – Scott Aaronson Feb 16 '13 at 6:27