Motivation for Heisenberg's modeling of observables

Question

What's the motivation for observables to be modeled by self-adjoint operators? I can't seem to find any place where this is laid out clearly. Maybe von Neumann's book is decent, but it's not satisfactory.

Background on Operational Physical Theories (TL;DR version, for detailed version, see K. Krauss' book States, Effects and Operations): The starting point of operational theories is collection of preparation instruments $S$, and measuring instruments $E$. Preparation instruments are characterised by the systems they prepare. We are interested in mathematically modeling these.

Measuring instruments are capable of undergoing changes upon interaction with the prepared system. The changes occurring are called ‘effects’. Any measuring equipment can be composed of simple yes-no equipment. Relative frequencies of occurrences gives us ‘operational statistics’ which are maps, $$\mu: S\times E\to \mathbb{R}, (\rho,X)\mapsto\mu(\rho\vert X)$$ which assign to each preparation procedure and measurement procedure, its frequency of going through the corresponding change. Accounting for the fact that we can combine preparation equipment, we should have, for a collection $\rho_i\in S$, and $\lambda_i \in [0,1]$ with $\sum_i\lambda_i =1$, $$\sum_i \lambda_i \rho_i\in S$$ or, it's a convex set. So, we can think of preparation procedures as linear maps on the space of effects of the form, $$X\mapsto \sum_i \alpha_i \mu(\rho_i\vert X).$$ This means each preparation procedure can be modelled as a linear functional on the space of effects, and similarly each effect can be viewed as a linear functional on the space of preparation procedures. So, each are vector spaces denote by $\mathcal{S}$ and $\mathcal{E}$ respectively.

Now we need more structure than mere vector space structure to do physics. In classical physics, the space of effects is modeled by a commutative algebra. In quantum theory, it's modeled by a C* algebra.

In classical physics, observables, are maps of the form, $$A: \mathcal{S}\to \mathcal{I}.$$ $\mathcal{I}$ is an indexing set, to which the values of the observables belong. By 20th century experiments showed the existence of continuous and discrete observables, and this meant discrete and continuous variables can't coexist.

The reasoning given is of the type: The existence of continuous physical quantities, i.e., a physical quantity whose values can be any real number, tells us that the cardinality of $\mathcal{S}$ should be at least uncountably infinite. However now, if there exists a discrete quantity, then some of the values should have infinite multiplicity in the state space i.e., infinitely many points in $\mathcal{S}$ are mapped to the same thing in $\mathcal{I}$. This model makes the coexistence of continuous quantities with discrete quantities.

See first 10 minutes in Connes' talk, Temps et aléa du quantique.

My problem is why not? Even for the case of momentum observable, on phase space, is a map, $$M:(q,p)\mapsto p.$$ The fiber has infinite cardinality for each value.

Heisenberg's idea of observable is completely new, and he thought of observables as self adjoint operators.

Can someone give me some nicer explanation as to why the old model for observables is bad, and Heisenberg's idea is better?

Answer 1

Sorry, for self answer, but I think this is what's happening. I don't know how this is related to Connes' explanation though.

Any measurement can be interpreted as a combination of 'yes-no' measurements. These 'yes-no' instruments can be used to build any general instrument. Suppose we have such an instrument, label its registration procedure by $R$. If the experiment is conducted a lot of times, we get a relative frequency of occurrence of 'yes'. Here 'yes' is an observable change in the instrument. It's hence an observable effect. To every preparation procedure $\rho$ and registration procedure $R_i$ there exists a probability $\mu(\rho, R_i)$ of occurrence of `yes' associated with the pair. $$(\rho,R_i)\longrightarrow \mu(\rho|R_i).$$ The numbers $\mu(\rho|R_i)$ are called operational statistics. Two completely different preparation procedures may give the same probabilities for all experiments $R$. Such preparation procedures must be considered equivalent. Such preparation procedures are called operationally equivalent preparations. An equivalence class of preparations procedures yielding the same result is called an ensemble. Ensembles are precursors to the notion of states.

The basic mathematical structure of ensembles and effects can be understood using purely mathematical reasons, without introducing any new physical law. Denote the class of ensembles by ${S}$ and the class of effects by ${E}$. The maps of interest to us are the following, $${S}\times {E}\xrightarrow{\mu}[0,1].$$ There may be two experiments that give the same probabilities for every ensemble. Such apparatuses must be considered equivalent. They are called operationally equivalent observable effects. An effect is the equivalence class of apparatuses yielding the same result. In general, a registration procedure $R$ for an experiment will have outcomes $\{R_i\}$. For an outcome, $R_i$ of the registration procedure $R$, denote the corresponding equivalence class of measurement procedures by $E_{R_i}$. Each outcome $R_i$ of the registration procedure corresponds to a functional $E_{R_i}$ called the effect of $R_i$ that acts on the ensemble of the system to yield the corresponding probability. $$E_{R_i}:\rho\mapsto E_{R_i}(\rho)=\mu(\rho|R_i).$$ Maps of interest to us will be those that assign to each of its outcomes $R_i$ its associated effect $E_{R_i}$. Since each ensemble fixes a probability distribution, we have, $$\mu_\rho:R_i\mapsto \mu_\rho(R_i)=\mu(\rho|R_i).$$ Accounting to the fact that preparation procedures can be combined to produce a mixed ensemble, the set of ensembles is taken to be a convex set. Since a mixture of ensembles corresponds to a convex combination of probabilities, each functional $E_{R_i}$ preserves the convex structure. Since two preparations giving the same result on every effect represent the same ensemble and two measurement procedures that cannot distinguish ensemble represent the same effect, ensembles and effects are mutually separating.

Denote by $\mathcal{S}$ the set of maps, $f:E\longrightarrow\mathbb{R}$ such that $f(X)=\sum_i \alpha_i \mu(\rho_i |X)$ and denote by $\mathcal{E}$ the set of maps, $g:S\longrightarrow\mathbb{R}$ such that $g(\rho)=\sum_i\beta_i \mu(\rho | R_i)$ where $\rho_i$ and $R_i$ are ensembles and effects respectively and $\alpha_i ,\beta_i\in\mathbb{R}$. Clearly $\mathcal{S}$ and $\mathcal{E}$ are real vector spaces. We can embed ensembles inside $\mathcal{S}$ with the map, $$\rho\longmapsto \mu_\rho,$$ and similarly embed effects inside $\mathcal{E}$ with the map, $$R_i\longmapsto E_{R_i}.$$ The bilinear map $\langle\cdot|\cdot\rangle : \mathcal{S}\times \mathcal{E}\to \mathbb{R}$ which coincides with $\mu$ is then uniquely determined. $\langle \mathcal{S}\:|\: \mathcal{E}\rangle$ becomes a dual pair. The completions of $\mathcal{S}$ and $\mathcal{E}$ will provide us the necessary mathematical structure for ensembles and effects. We will denote $\langle \cdot |\cdot \rangle$ by $\mu$. This gives us a vector space structure for space of effects and ensembles. Now we want to give more structure on these.

An observable corresponds to a collection of effects that can be measured by a measuring equipment. So, this should correspond to a Boolean lattice homomorphism, $$A:\Sigma_A\to \mathcal{E}$$ Here $\Sigma_A$ is some Boolean lattice. When measurements are performed, the total probability of occurrence of the effects of the observable should be $1$.

By the end of the nineteenth century, it was clear that elementary processes obeyed some ‘discontinuous’ laws. That's to say, there exist observables whose collection of effects form discrete sets, and also observables whose collection of effects form a continuous set. If the collection of effects of an observable is labeled by a discrete set $\{R_i\}_{i\in\mathbb{Z}}$, each ensemble $\rho$ corresponding to some preparation procedure gives rise to a function, \begin{align*} \mathbb{Z} \xrightarrow{\mu_\rho} [0,1] \end{align*} Since preparation procedures are independent of which observable is measured, each ensemble should also give us a map, $$\mathcal{B}(\mathbb{R}) \xrightarrow{\mu_\rho} [0,1]$$ corresponding to continuous observable. Corresponds to the fact that the total probability of occurance of one of the effects of an observable in the collection $\{R_i\}$ should be $1$, the above described functions should be summable of integrable.

Before quantum theory, observables were modeled as functions on a manifold. Each state corresponds to a point in the manifold. So each state can be viewed as an evaluation map. This is however problematic if we take discrete observables into account. In such a case, certain values of the observable will have infinite multiplicity, and hence will not be summable. Hence, the pre-quantum modeling of observables cannot unify discrete and continuous physical variables. Heisenberg's radical idea was to start rethinking how we should model observables themselves. Heisenberg used linear operators as models for observables, and the values of the observables corresponded to the eigenvalues of these operators. von Neumann took away from this the following idea, instead of considering the relation between discrete space and continuous space, von Neumann compared the relationship between the functions on the discrete space and continuous space. The space of square-integrable functions on $\mathbb{R}$ is isomorphic to the space of square summable sequences, which are functions on $\mathbb{Z}$. This isomorphism allows us to develop a unified mathematical model for observables and states. The square summable functions correspond to ensembles, and the squares correspond to probabilities. The necessary structure for the abstract mathematical framework of quantum theory is found in Hilbert spaces and operator algebras.